Dynamic Scalable Event Chain Tracing API
An API and implementation to install debug probes into a parallel application
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
An API and implementation to install debug probes into a parallel application
STAT is a light weight debugging tool that gathers and merges stack traces from all of the processes in a parallel application. STAT uses the MRNet tree based overlay network to broadcast commands from the tool front-end to the STAT daemons and for the front-end to gather the traces from the STAT daemons. As the traces propagate through the MRNet network tree, they are merged across all tasks to fonn a single call prefix tree. The call prefix tree can be examined to identify tuks with similar function call patterns and to delineate a small set of equivalence classes. A representative task from each of these classes can then be fed into a full feature debugger like TotalView for root cause analysis.
loupe is a PMPl-based library to profile MPI routines in a parallel application.
This paper describes a kernel scheduling algorithm that is based on coscheduling principles and that is intended for parallel applications running on 1000 cores or more. Experimental results for a Linux implementation on a Cray XT5 machine are presented. The results indicate that Linux is a suitable operating system for this new scheduling scheme, and that this design provides a dramatic improvement in scaling performance for synchronizing collective operations at scale.
Darshan is a production-quality I/O characteriza tion tool that captures and summarizes the I/O behavior of parallel applications. It records a variety of information with minimal overhead, including access patterns, number of files accessed, and the amount of time consumed by I/O routines. Darshan’s lightweight design makes it suitable for full-time deployment for workload characterization of large HPC systems.
This report is a DARPA funded Work For Others (WFO) evaluation of the Parallel Tools Platform (PTP) version 5.0.3 (Released October 18, 2011). PTP is currently being developed and supported as an open source product managed by Project Lead Greg Watson and team at IBM plus several other contributors. The tool has been in development since 2005 with a goal of providing an infrastructure for the integration of high performance computing tools. Basic functionality such as source code editing, compiling, launching, and debugging, plus more advanced features including performance monitoring and deployment have been bundled into an Eclipse Integrated Development Environment (IDE) plug-in to give parallel application programmers the features and improved productivity gained from using a common, well established development platform. The evaluation encompasses the basic aspects of using PTP, beginning with the installation and configuration of the plug-in, followed by code development and debugging. Each aspect is evaluated based on the expectations of my experience as an Eclipse user and is presented in the following format: • Feature description • User Experience • Evaluation The feature description is based on the provided documentation for PTP and the ‘generic’ expectations of what the feature should provide. The user experience reveals the actual use and exercise of the feature – specifically in the context and perspective of the evaluation platform provided by Lawrence Livermore National Laboratory. The evaluation of each feature includes the overall outcome of the user experience, along with the expectations that were not met, met, or exceeded along with any issues that were encountered.
Processors with multiple (or many) cores and shared memory are becoming ubiquitous across the com puting spectrum. MPI, the current de facto program ming model for scalable parallel applications, enforces copies between source and target processes and thus can not fully utilize shared memory and cache architectures of modern machines. To enable MPI-based programs to more fully exploit features of multi- and many-core architectures, we present a compiler-based transformation that transforms MPI processes into threads and fuses message serialization and deserialization loops such that send and receive calls can be replaced by direct memory accesses. Our compiler replaces most of the MPI communication functions with direct load/store accesses and our runtime provides a threaded MPI implementation to handle the remaining functions.
Load imbalance can lead to wasted CPU hours especially when running on a large number of processors. Achieving the best parallel efficiency for a program requires optimal load balancing which is a NP-hard problem. Charm++, a migratable objects based programming model, provides a measurement-based dynamic load balancing frame work. This paper explores the use of graph partitioning algorithms, traditionally used for partitioning physical domains/meshes, for measurement based dynamic load balancing of parallel applications. In particular, we present repartitioning methods developed in a graph partitioning toolbox called Scotch that consider the previous mapping to minimize migration costs. We also discuss a new imbalance reduction algorithm for graphs with irregular load distributions. We compare several load balancing algorithms using a micro-benchmark and the NAS BT multi-zone bench mark. New algorithms developed in Scotch lead to better performance compared to other existing partitioners, both in terms of the application execution time and fewer number of objects migrated.
We report on the use and design of a portable, extensible performance data collection tool motivated by modeling needs of the high performance computing systems co-design com- munity. The lightweight performance data collectors with Eiger support is intended to be a tailorable tool, not a shrink-wrapped library product, as pro ling needs vary widely. A single code markup scheme is reported which, based on compilation ags, can send perfor- mance data from parallel applications to CSV les, to an Eiger mysql database, or (in a non-database environment) to at les for later merging and loading on a host with mysql available. The tool supports C, C++, and Fortran applications.
Future performance improvements for microprocessors have shifted from clock frequency scaling towards increases in on-chip parallelism. Performance improvements for a wide variety of parallel applications require domain-decomposition of data arrays from a contiguous arrangement in memory to a tiled layout for on-chip L1 data caches and scratchpads. How- ever, DRAM performance suffers under the non-streaming access patterns generated by many independent cores. We propose collective memory scheduling (CMS) that actively takes control of collective memory transfers such that requests arrive in a sequential and predictable fashion to the memory controller. CMS uses the hierarchically tiled arrays formal- ism to compactly express collective operations, which greatly improves programmability over conventional prefetch or list- DMA approaches. CMS reduces application execution time by up to 32% and DRAM read power by 2.2×, compared to a baseline DMA architecture such as STI Cell.
This project concentrated on various ways to improve the measurement and tuning large-scale parallel applications. This project was supplement to the project DE-FC0206ER25763 (“Performance Engineering Research Center”). The research conducted during this project is summarized in this report. The complete details of the work are available in the ten publications listed at the end of the report. It also supported the Ph.D. studies of three students and one research scientist.
This project seeks to incorporate OpenFlow-style Software Defined Networking capability into existing High Performance Computing (HPC) interconnect technology, InfiniBand, and to extend such capability beyond one domain. We develop SDN-enhanced InfiniBand that supports OpenFlow functionality for HPC interconnects, investigate the required techniques and the potential benefits for applying SDN on different types of HPC interconnects, and develop software to control the network and expose SDN capability to external networks through the OpenFlow interface. We have also developed techniques to evaluate communication characteristics of parallel applications, which determines the workload on interconnects, investigated newly proposed Interconnection network technologies including new network topologies and routing schemes, and designed novel interconnect routing and modeling techniques. Major activities and findings are summarized as follows.
Several variations of cache-based checkpointing for rollback error recovery in shared-memory multiprocessors have been recently developed. By modifying the cache replacement policy, these techniques use the inherent redundancy in the memory hierarchy to periodically checkpoint the computation state. Three schemes, different in the manner in which they avoid rollback propagation, are evaluated. By simulation with address traces from parallel applications running on an Encore Multimax shared-memory multiprocessor, the performance effect of integrating the recovery schemes in the cache coherence protocol are evaluated. The results indicate that the cache-based schemes can provide checkpointing capability with low performance overhead but uncontrollable high variability in the checkpoint interval.
The low cost and availability of clusters of workstations have lead researchers to re-explore distributed computing using independent workstations. This approach may provide better cost/performance than tightly coupled multiprocessors. In practice, this approach often utilizes wasted cycles to run parallel jobs. The feasibility of such a non-dedicated parallel processing environment assuming workstation processes have preemptive priority over parallel tasks is addressed. An analytical model is developed to predict parallel job response times. Our model provides insight into how significantly workstation owner interference degrades parallel program performance. A new term task ratio, which relates the parallel task demand to the mean service demand of nonparallel workstation processes, is introduced. It was proposed that task ratio is a useful metric for determining how large the demand of a parallel applications must be in order to make efficient use of a non-dedicated distributed system.
Significant progress has been made in hardware and software technologies, as a result the use of efficient standard message passing communication calls in parallel application codes has greatly assisted in reducing execution times. In addition, the introduction of OpenMP, an industrial standard for shared memory programming with directives addresses the issue of portability and also provides a means to deliver respectable performance. In general however, the manual parallelisation of a user's application code using either message passing or directives is still a very costly and error-prone process.
In this paper we seek to demonstrate the importance of studying the effect of changes in execution environment parameters, on parallel applications executed on state-of-the-art multiprocessors. A comprehensive methodology for event-based analysis of program behavior is introduced. This methodology is used to study the performance significance of various system parameters such as processor speed, message-buffer size, buffer copy speed, network bandwidth, communication latency, interrupt overheads and other system parameters. With the help cf a few CFD examples, we illustrate the use of our technique in determining suitable parameter values of the execution environment for three applications. We also demonstrate how this approach can be used to predict performance across architectures and illustrate the use of visual and profile-like feedback to expose the effect of system parameters changes on the performance of specific applications module.
Single-slope analog-to-digital converters (ADCs) are particularly useful for onchip digitization in focal plane arrays (FPAs) because of their inherent monotonicity, relative simplicity, and efficiency for column-parallel applications, but they are comparatively slow. Squareroot encoding can allow the number of code values to be reduced without loss of signal-to-noise ratio (SNR) by keeping the quantization noise just below the signal shot noise. This encoding can be implemented directly by using a quadratic ramp. The reduction in the number of code values can substantially increase the quantization speed. However, in an FPA, the fixed pattern noise (FPN) limits the use of small quantization steps at low signal levels. If the zero-point is adjusted so that the lowest column is onscale, the other columns, including those at the center of the distribution, will be pushed up the ramp where the quantization noise is higher. Additionally, the finite frequency response of the ramp buffer amplifier and the comparator distort the shape of the ramp, so that the effective ramp value at the time the comparator trips differs from the intended value, resulting in errors. Allowing increased settling time decreases the quantization speed, while increasing the bandwidth increases the noise. The FPN problem is solved by breaking the ramp into two portions, with some fraction of the available code values allocated to a linear ramp and the remainder to a quadratic ramp. To avoid large transients, both the value and the slope of the linear and quadratic portions should be equal where they join. The span of the linear portion must cover the minimum offset, but not necessarily the maximum, since the fraction of the pixels above the upper limit will still be correctly quantized, albeit with increased quantization noise. The required linear span, maximum signal and ratio of quantization noise to shot noise at high signal, along with the continuity requirement, determines the number of code values that must be allocated to each portion. The distortion problem is solved by using a lookup table to convert captured code values back to signal levels. The values in this table will be similar to the intended ramp value, but with a correction for the finite bandwidth effects. Continuous-time comparators are used, and their bandwidth is set below the step rate, which smoothes the ramp and reduces the noise. No settling time is needed, as would be the case for clocked comparators, but the low bandwidth enhances the distortion of the non-linear portion. This is corrected by use of a return lookup table, which differs from the one used to generate the ramp. The return lookup table is obtained by calibrating against a stepped precision DC reference. This results in a residual non-linearity well below the quantization noise. This method can also compensate for differential non-linearity (DNL) in the DAC used to generate the ramp. The use of a ramp with a combination of linear and quadratic portions for a single-slope ADC is novel. The number of steps is minimized by keeping the step size just below the photon shot noise. This in turn maximizes the speed of the conversion. High resolution is maintained by keeping small quantization steps at low signals, and noise is minimized by allowing the lowest analog bandwidth, all without increasing the quantization noise. A calibrated return lookup table allows the system to maintain excellent linearity.
The current trend of multi-core and multi-processor computing is causing a paradigm shift from inherently sequential to highly concurrent and parallel applications. Certain thread interleavings, data input values, or combinations of both often cause errors in the system. Systematic verification techniques such as explicit state model checking and symbolic execution are extensively used to detect errors in such systems [7, 9]. Explicit state model checking enumerates possible thread schedules and input data values of a program in order to check for errors [3, 9]. To partially mitigate the state space explosion from data input values, symbolic execution techniques substitute data input values with symbolic values [5, 7, 6]. Explicit state model checking and symbolic execution techniques used in conjunction with exhaustive search techniques such as depth-first search are unable to detect errors in medium to large-sized concurrent programs because the number of behaviors caused by data and thread non-determinism is extremely large. We present an overview of abstraction-guided symbolic execution for concurrent programs that detects errors manifested by a combination of thread schedules and data values [8]. The technique generates a set of key program locations relevant in testing the reachability of the target locations. The symbolic execution is then guided along these locations in an attempt to generate a feasible execution path to the error state. This allows the execution to focus in parts of the behavior space more likely to contain an error.