Engineering Papers⌕ Search

Engineering topics

Lofstead, J.

Publications and source records attributed to Lofstead, J..

Exascale Storage Systems the SIRIUS Way

As the exascale computing age emerges, data related issues are becoming critical factors that determine how and where we do computing. Popular approaches used by traditional I/O solution and storage libraries become increasingly bottlenecked due to their assumptions about data movement, re-organization, and storage. While, new technologies, such as "burst buffers", can help address some of the short-term performance issues, it is essential that we reexamine the underlying storage and I/O infrastructure to effectively support requirements and challenges at exascale and beyond. In this paper we present a new approach to the exascale Storage System and I/O (SSIO), which is based on allowing users to inject application knowledge into the system and leverage this knowledge to better manage, store, and access large data volumes so as to minimize the time to scientific insights. Central to our approach is the distinction between the data, metadata, and the knowledge contained therein, transferred from the user to the system by describing "utility" of data as it ages.

97 MATHEMATICS AND COMPUTING↗

Experiences Applying Data Staging Technology in Unconventional Ways

Several efforts have shown the potential of using additional compute-area resources to enhance the IO path to storage. Efforts like data staging, IO forwarding, and similar techniques can accelerate IO performance and reduce the impact of IO time to a compute application. Hybrid staging enhanced this path by adding processing functionality to locations along the data path to storage. While these efforts have been effective, they have taken a somewhat limited view of the potential benefits using some additional compute resources can offer both to enhance a compute application as well as to offering a way to exploit HPC-style resources for non-traditional tasks. Over the last few years, we have been experimenting with the potential for other sorts of activities using a staging style approach to add or enable new functionality. The efforts in this area have yielded a collection of small projects that yield some insights into both the potential and limitations of this approach for both achieving exascale computing and for enabling alternative uses for HPC resources.

Computer Science↗

EFFIS: An End-to-end Framework for Fusion Integrated Simulation

The purpose of the Fusion Simulation Project is to develop a predictive capability for integrated modeling of magnetically confined burning plasmas. In support of this mission, the Center for Plasma Edge Simulation has developed an End-to-end Framework for Fusion Integrated Simulation (EFFIS) that combines critical computer science technologies in an effective manner to support leadership class computing and the coupling of complex plasma physics models. We describe here the main components of EFFIS and how they are being utilized to address our goal of integrated predictive plasma edge simulation.

Computer Science↗

PreDatA - Preparatory Data Analytics on Peta-Scale Machines

Peta-scale scientific applications running on High End Computing (HEC) platforms can generate large volumes of data. For high performance storage and in order to be useful to science end users, such data must be organized in its layout, indexed, sorted, and otherwise manipulated for subsequent data presentation, visualization, and detailed analysis. In addition, scientists desire to gain insights into selected data characteristics 'hidden' or 'latent' in the massive datasets while data is being produced by simulations. PreDatA, short for Preparatory Data Analytics, is an approach for preparing and characterizing data while it is being produced by the large scale simulations running on peta-scale machines. By dedicating additional compute nodes on the peta-scale machine as staging nodes and staging simulation's output data through these nodes, PreDatA can exploit their computational power to perform selected data manipulations with lower latency than attainable by first moving data into file systems and storage. Such in-transit manipulations are supported by the PreDatA middleware through RDMA-based data movement to reduce write latency, application-specific operations on streaming data that are able to discover latent data characteristics, and appropriate data reorganization and metadata annotation to speed up subsequent data access. As a result, PreDatA enhances the scalability and flexibility of current I/O stack on HEC platforms and is useful for data pre-processing, runtime data analysis and inspection, as well as for data exchange between concurrently running simulation models. Performance evaluations with several production peta-scale applications on Oak Ridge National Laboratory's Leadership Computing Facility demonstrate the feasibility and advantages of the PreDatA approach.

99 GENERAL AND MISCELLANEOUS↗

Whole-volume integrated gyrokinetc simulation of plasma turbulence in realistic diverted-tokamak geometry

Performance prediction for ITER is based upon the ubiquitous experimental observation that the plasma energy confinement in the device core is strongly coupled to the edge confinement for an unknown reason. The coupling time-scale is much shorter than the plasma transport time-scale. In order to understand this critical observation, a multi-scale turbulence-neoclassical simulation of integrated edge-core plasma in a realistic diverted geometry is a necessity, but has been a formidable task. Thanks to the recent development in high performance computing, we have succeeded in the integrated multiscale gyrokinetic simulation of the ion-temperature-gradient driven turbulence in realistic diverted tokamak geometry for thefirst time. It is found that modification of the self-organized criticality in the core plasma by nonlocal core-edge coupling of ITG turbulence can be responsible for the core-edge confinement coupling.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Advanced simulation of electron heat transport in fusion plasmas

Electron transport in burning plasmas is more important since fusion products first heat electrons. First-principles simulations of electron turbulence are much more challenging due to the multi-scale dynamics of the electron turbulence, and have been made possible by close collaborations between plasma physicists and computational scientists. The GTC simulations of collisionless trapped electron mode (CTEM) turbulence show that the electron heat transport exhibits a gradual transition from Bohm to gyroBohm scaling when the device size is increased. The deviation from the gyroBohm scaling can be induced by large turbulence eddies, turbulence spreading, and non-diffusive transport processes. Analysis of radial correlation function shows that CTEM turbulence eddies are predominantly microscopic but with a significant tail in the mesoscale. A comprehensive analysis of kinetic and fluid time scales shows that zonal flow shearing is the dominant decorrelation mechanism. The mesoscale eddies result from a dynamical process of linear streamers breaking by zonal flows and merging of microscopic eddies. The radial profile of the electron heat conductivity only follows the profile of fluctuation intensity on a global scale, whereas the ion transport tracks more sensitively the local fluctuation intensity. This suggests the existence of a nondiffusive component in the electron heat flux, which arises from the ballistic radial E x B drift of trapped electrons due to a combination of the presence of mesoscale eddies and the weak de-tuning of the toroidal precessional resonance that drives the CTEM instability. On the other hand, the ion radial excursion is not affected by the mesoscale eddies due to a parallel decorrelation, which is not operational for the trapped electrons because of a bounce averaging process associated with the electron fast motion along magnetic field lines. The presence of the nondiffusive component raises question on the applicability of the usual quasilinear theory for the CTEM electron transport. This is in contrast to the good agreement between the quasilinear transport theory and simulation results of the electron heat transport in electron temperature gradient (ETG) turbulence, which is regulated by a wave-particle decorrelation. Therefore, the transport in the CTEM turbulence is a fluid-like eddy mixing process even though the linear CTEM instability is driven by a kinetic resonance. In contrast, a kinetic process dominates the transport in the ETG turbulence, which is characterized by macroscopic streamers.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Petascale IO Using The Adaptable IO System

ADIOS, the adaptable IO system, has demonstrated excellent scalability to 29,000 cores. With the introduction of the XT5 upgrades to Jaguar, new optimizations are required to successfully reach 140,000+ cores. This paper explains the techniques employed and shows the performance levels attained.

97 MATHEMATICS AND COMPUTING↗

Adaptable Metadata Rich IO Methods for Portable High Performance IO

Since IO performance on HPC machines strongly depends on machine characteristics and configuration, it is important to carefully tune IO libraries and make good use of appropriate library APIs. For instance, on current petascale machines, independent IO tends to outperform collective IO, in part due to bottlenecks at the metadata server. The problem is exacerbated by scaling issues, since each IO library scales differently on each machine, and typically, operates efficiently to different levels of scaling on different machines. With scientific codes being run on a variety of HPC resources, efficient code execution requires us to address three important issues: (1) end users should be able to select the most efficient IO methods for their codes, with minimal effort in terms of code updates or alterations; (2) such performance-driven choices should not prevent data from being stored in the desired file formats, since those are crucial for later data analysis; and (3) it is important to have efficient ways of identifying and selecting certain data for analysis, to help end users cope with the flood of data produced by high end codes. This paper employs ADIOS, the ADaptable IO System, as an IO API to address (1)-(3) above. Concerning (1), ADIOS makes it possible to independently select the IO methods being used by each grouping of data in an application, so that end users can use those IO methods that exhibit best performance based on both IO patterns and the underlying hardware. In this paper, we also use this facility of ADIOS to experimentally evaluate on petascale machines alternative methods for high performance IO. Specific examples studied include methods that use strong file consistency vs. delayed parallel data consistency, as that provided by MPI-IO or POSIX IO. Concerning (2), to avoid linking IO methods to specific file formats and attain high IO performance, ADIOS introduces an efficient intermediate file format, termed BP, which can be converted, at small cost, to the standard file formats used by analysis tools, such as NetCDF and HDF-5. Concerning (3), associated with BP are efficient methods for data characterization, which compute attributes that can be used to identify data sets without having to inspect or analyze the entire data contents of large files.

97 MATHEMATICS AND COMPUTING↗

I/O through High Performance Data Services

The conclusions of this paper are that: (1) moving output data processing to a separate processing area has performance advantages; (2) management of data movement results in large performance gains; (3) the Data Service abstraction enables the development of data staging operations; and (4) using the data service model, we can create adaptive resource aware storage backends.

97 MATHEMATICS AND COMPUTING↗

...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats

As HPC applications run on increasingly high process counts on larger and larger machines, both the frequency of checkpoints needed for fault tolerance and the resolution and size of Data Analysis Dumps are expected to increase proportionally. In order to maintain an acceptable ratio of time spent performing useful computation work to time spent performing I/O, write bandwidth to the underlying storage system must increase proportionally to this increase in the checkpoint and computation size. Unfortunately, popular scientific self-describing file formats such as netCDF and HDF5 are designed with a focus on portability and flexibility. Extra care and careful crafting of the output structure and API calls is required to optimize for write performance using these APIs. To provide sufficient write bandwidth to continue to support the demands of scientific applications, the HPC community has developed a number of I/O middleware layers, that structure output into write-optimized file formats. However, the obvious concern with any write optimized file format would be a corresponding penalty on reads. In the log-structured filesystem, for example, a file generated by random writes could be written efficiently, but reading the file back sequentially later would result in very poor performance. Simulation results require efficient read-back for visualization and analytics, and though most checkpoint files are never used, the efficiency of a restart is very important in the face of inevitable failures. The utility of write speed improving middleware would be greatly diminished if it sacrificed acceptable read performance. In this paper we examine the read performance of two write-optimized middleware layers on large parallel machines and compare it to reading data natively in popular file formats. The two I/O middleware layer examined in this paper are the Adaptable IO System (ADIOS), a library-based approach developed at Oak Ridge National Laboratory to provide a high-level IO API that can be used in place of netCDF or HDF5 to do much more aggressive write-behind and efficient reordering of data locations within the file; and the Parallel Log-structured Filesystem (PLFS), a stackable FUSE filesystem approach developed at Los Alamos National Laboratory that decouples concurrent writes to improve the speed of checkpoints. Since ADIOS is an I/O componentization that affords selection of different I/O methods at or during runtime, through a single API, users can have access to MPI-IO, Posix-IO, HDF5, netCDF, and staging methods. The ADIOS BP file format is a new log-file format that has a superset of the features of both HDF5 and netCDF, but is designed to be portable and flexible while being optimized for writing. PLFS, on the other hand, is mounted as a stackable filesystem on top of an existing parallel filesystem. Reads or writes to the PLFS filesystem are transparently translated into operations on per-process log files stored in the underlying parallel filesystem. Since PLFS performs this translation without application modification, users can write in HDF5, netCDF, or app-specific file formats and PLFS will store the writes in a set of efficiently written log-formatted files, while presenting the user with a logical 'flat' file on reads. Despite their different approaches, the commonality behind both of these middleware systems is that they both write to a log file format. As shown in previous publications, writes are fully optimized in both systems, sometimes resulting in 100x improvements over writing data in popular file formats. But as mentioned above, writing is only one part of the story. In this paper we examine the read performance of our middleware layers on large parallel machines and compare these to reading data either natively or from other popular file formats. We compare the reading performance in two different scenarios: (1) Reading back restarts from the same number of processors as wrote the data and (2) Reading back restart data from a different number of processors. We observe that not only can write-optimized I/O middleware be built to not penalize read speeds, but for important workloads, techniques that improve write performance can, perhaps counterintuitively, improve read speeds over reading from a contiguously organized file format. In the remainder of this paper, we investigate this further through case studies of PLFS and ADIOS on simulation checkpoint restarts.

99 GENERAL AND MISCELLANEOUS↗