Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “hdf5”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

PSIP For HDF5 Pilot Project (Final Report)

Productivity and Sustainability Improvement Planning (PSIP) is a lightweight, incremental and iterative approach (much in the same spirit as Agile methodologies) for making routine software process improvements in software projects. It is designed to be easily applied in existing development workflows. Quoting from a November 2019 workshop paper describing PSIP. PSIP breaks from classic software process improvement approaches such as CMM(I), SPICE, ISO 9000 or Six Sigma, in that it trades comprehensive standards and certification-driven assessment for self-defined, internally driven goals. It does, however, carry forward such ideas as having staged models of improvement (like CMM(I)) in the form of progress tracking cards. Additionally, PSIP is more aligned with lean and agile methods; it adopts their emphasis on iterative improvement and continuous learning. At its core, PSIP is an instantiation of the Plan-Do-Check-Act management cycle (PDCA, also known as Plan-Do-Study-Adjust) which provides the foundation for much of the modern software process improvement literature. This situates PSIP within a constellation of bottom-up, inductive software process improvement methods. PSIP is designed around the notion that already overburdened teams can define and carry out a series of small, incremental steps of progression towards improvement goals without significant (there will be some, but the goal is to avoid significant) disruption to ongoing development activities. A key enabling tool in PSIP is the use of Progress Tracking Cards (PTCs) which define the steps of progression towards a given improvement goal. This is the theory of PSIP. The PSIP for HDF5 project was aimed at putting PSIP into practice with the purpose of evaluating its effectiveness in planning and facilitating quality and process improvements in a scientific software project as well as its associated artifacts. The HDF5 project was chosen as a use case to evaluate PSIP for several reasons. First, NNSA labs and LLNL in particular have a keen interest in how HDF5 quality impacts its uptake and sustainability as a community adopted and supported code. Next, HDF5 is a foundational library, a key substrate in the HPC/CSE software stack, and any improvements there realized through this contract will have benefits to many DOE applications depending on it. HDF5 also represents an older, legacy code with technical debts to pay down. These characteristics are similar to many NNSA and even some ECP code projects. But, because HDF5 is an I/O library, it represents a simpler use case within which to study PSIP than a full-fledged and significantly more complex PDE simulation code. We believe these attributes make HDF5 an ideal use case for evaluating PSIP.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

GPU Direct I/O with HDF5

Exascale HPC systems are being designed with accelerators, such as GPUs, to accelerate parts of applications. In machine learning workloads as well as large-scale simulations that use GPUs as accelerators, the CPU (or host) memory is currently used as a buffer for data transfers between GPU (or device) memory and the file system. If the CPU does not need to operate on the data, then this is sub-optimal because it wastes host memory by reserving space for duplicated data. Furthermore, this “bounce buffer” approach wastes CPU cycles spent on transferring data. A new technique, NVIDIA GPUDirect Storage (GDS), can eliminate the need to use the host memory as a bounce buffer. Thereby, it becomes possible to transfer data directly between the device memory and the file system. This direct data path shortens latency by omitting the extra copy and enables higher-bandwidth. To take full advantage of GDS in existing applications, it is necessary to provide support with existing I/O libraries, such as HDF5 and MPI-IO, which are heavily used in applications. In this paper, we describe our effort of integrating GDS with HDF5, the top I/O library at NERSC and at DOE leadership computing facilities. We design and implement this integration using a HDF5 Virtual File Driver (VFD). The GDS VFD provides a file system abstraction to the application that allows HDF5 applications to perform I/O without the need to move data between CPUs and GPUs explicitly. We compare performance of the HDF5 GDS VFD with explicit data movement approaches and demonstrate superior performance with the GDS method.

Ravi, J↗
DOE OSTI2020

HDF5-Version 1.12.0

The HDF5 library and file format implement the HDF5 data model for storing and managing data. This implementation supports an unlimited variety of datatypes and is designed for flexible and efficient I/O and for high volume and complex data. It is highly portable and extensible, allowing applications to evolve in their use of HDF5. The HDF5 Technology suite includes tools and applications for managing, manipulating, viewing, and analyzing data in the HDF5 format and other implementations.

The HDF Group, None↗
DOE OSTI2020

Interfacing HDF5 with a scalable object‐centric storage system on hierarchical storage

Summary Object storage technologies that take advantage of multitier storage on HPC systems are emerging. However, to use these technologies at present, applications have to be modified significantly from current I/O libraries. HDF5, a widely used I/O middleware on HPC systems, provides a virtual object layer (VOL) that allows applications to connect to different storage mechanisms transparently without requiring significant code modifications. We recently designed the proactive data containers (PDC) object‐centric storage system that provides the capabilities of transparent, asynchronous, and autonomous data movement taking advantage of multiple storage tiers—a decision that has so far been left upon the user on most current systems. To enable PDC's features through HDF5 without modifying application codes, we have developed an HDF5 VOL connector that interfaces with PDC. We present in this article the connector interface and evaluate its performance on Cori, a Cray XC40 supercomputer located at the National Energy Research Scientific Computing Center (NERSC). Our evaluation demonstrates up to an 8× improvement compared with HDF5 that has the most recent optimizations.

Mu, Jingqing↗
DOE OSTI2021

Battle of the Defaults: Extracting Performance Characteristics of HDF5 under Production Load

Popular parallel I/O libraries, such as HDF5, provide tuning parameters to obtain superior performance. However, the selection of effective parameters on production systems is complex due to the interdependence of I/O software and file system layers. Hence, application developers typically use the default parameters and often experience poor I/O performance. This work conducts a benchmarking-based analysis on the HDF5 behaviors with a wide variety of I/O patterns to extract performance characteristics under the production workload. To make the analysis well controlled, we exercise I/O benchmarks on POSIX-IO, MPI-IO, and HDF5 using the same I/O patterns and in the same jobs. To address high performance variability in production environments, we repeat the benchmarks across I/O patterns, storage devices, and time intervals. Based on the results, we identified consistent HDF5 behaviors that appropriate configurations and operations on dataset layout and file-metadata placement can improve performance significantly. We apply our findings and evaluate the tuned I/O library on two supercomputers: Summit and Cori. The results show that our tuned parameters can achieve more than 10× I/O performance speedup than that with default parameters on both systems, suggesting the effectiveness, stability, and generality of our solution.

Xie, Bing↗
DOE OSTI2024

h5bench: A unified benchmark suite for evaluating HDF5 I/O performance on pre‐exascale platforms

Summary Parallel I/O is a critical technique for moving data between compute and storage subsystems of supercomputers. With massive amounts of data produced or consumed by compute nodes, high‐performant parallel I/O is essential. I/O benchmarks play an important role in this process; however, there is a scarcity of I/O benchmarks representative of current workloads on HPC systems. Toward creating representative I/O kernels from real‐world applications, we have created h5bench , a set of I/O kernels that exercise hierarchical data format version 5 (HDF5) I/O on parallel file systems in numerous dimensions. Our focus on HDF5 is due to the parallel I/O library's heavy usage in various scientific applications running on supercomputing systems. The various tests benchmarked in the h5bench suite include I/O operations (read and write), data locality (arrays of basic data types and arrays of structures), array dimensionality (one‐dimensional arrays, two‐dimensional meshes, three‐dimensional cubes), I/O modes (synchronous and asynchronous). In this paper, we present the observed performance of h5bench executed along several of these dimensions on existing supercomputers (Cori and Summit) and pre‐exascale platforms (Perlmutter, Theta, and Polaris). h5bench measurements can be used to identify performance bottlenecks and their root causes and evaluate I/O optimizations. As the I/O patterns of h5bench are diverse and capture the I/O behaviors of various HPC applications, this study will be helpful to the broader supercomputing and I/O community.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

HDF5 in the exascale era: Delivering efficient and scalable parallel I/O for exascale applications

Accurately modeling real-world systems requires scientific applications at exascale to generate massive amounts of data and manage data storage efficiently. However, parallel input and output (I/O) faces challenges due to new application workflows and the state-of-the-art memory, interconnect, and storage architectures considered in exascale designs. The storage hierarchy has expanded with node-local persistent memory, solid-state storage, and traditional disk and tape-based storage, thus requiring efficiency at each layer and much more efficient data movement among these layers. This paper discusses how the ExaHDF5 project improved the I/O performance and data management for exascale architectures by enhancing HDF5, a widely used parallel I/O library. The team developed an Asynchronous I/O Virtual Object Layer (VOL) connector that allowed overlapping I/O with computation. They also created a Cache VOL to complement asynchronous I/O by incorporating fast storage layers, such as burst buffer and node-local storage, into the parallel I/O workflow through caching and staging data. Additionally, the team enabled data aggregation and I/O at the node level by using a Subfiling Virtual File Driver (VFD). To demonstrate superior I/O performance with HDF5 at exascale, the ExaHDF5 team collaborated with several exascale applications. In this paper, we show I/O performance improvements for three applications: Cabana (a particle-based simulation library), EQSIM (a regional earthquake simulation software), and E3SM (a climate system modeling library).

Asynchronous I/Ol↗
DOE OSTI2024

Delta-Rice: A HDF5 Compression Plugin optimized for Digitized Detector Data

Delta-Rice is an HDF5 (The HDF Group et al., 2020) filter plugin that was developed to compress digitized detector signals recorded by the Nab experiment (Fry et al., 2019), a fundamental neutron physics experiment. This is a two-step process where incoming data is passed through a pre-processing filter and then compressed with Rice coding. A routine for determining the optimal pre-processing filter for a dataset is provided along with an example GPU deployment. When applied to data collected by the Nab data acquisition system, this method produced output files 29% their initial size, and was able to do so with an average read/write throughput in excess of 2 GB/s on a single CPU. Compared to the widely used Gzip compression routine, Delta-Rice reduces the file size by 33% more with over an order of magnitude increase in read/write throughput. Delta-Rice is available on CPU to users through the HDF5 library.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

A case study on parallel HDF5 dataset concatenation for high energy physics data analysis

In High Energy Physics (HEP), experimentalists generate large volumes of data that, when analyzed, helps us better understand the fundamental particles and their interactions. This data is often captured in many files of small size, creating a data management challenge for scientists. In order to better facilitate data management, transfer, and analysis on large scale platforms, it is advantageous to aggregate data further into a smaller number of larger files. However, this translation process can consume significant time and resources, and if performed incorrectly the resulting aggregated files can be inefficient for highly parallel access during analysis on large scale platforms. In this paper, we present our case study on parallel I/O strategies and HDF5 features for reducing data aggregation time, making effective use of compression, and ensuring efficient access to the resulting data during analysis at scale. We focus on NOvA detector data in this case study, a large-scale HEP experiment generating many terabytes of data. Here, the lessons learned from our case study inform the handling of similar datasets, thus expanding community knowledge related to this common data management task.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2020

Independent Metadata Update VOL Connector for HDF5 (IMM VOL) v0.1

We have built a new framework that enables independent metadata updates within HDF5 files. Our framework includes the following components: an overlay network that's optimized for 1-to-All and All-to-All communication; root-less/server-less Bcast and AllReduce operations for reduced synchronization; a distributed consensus-making platform; an execution order keeping system, and a progress engine for generic threadless concurrence handling.

Li, Tonglin↗
DOE OSTI2021

The Story of HDF5 in High Energy Physics

For several years now, Fermilab has been investigating the use of HDF5 for large-scale analysis of experimental high energy physics (HEP) data.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

Fission Matrix Processing Using the MCNP6.3 HDF5 Restart File

This paper describes an approach to interrogating the fission matrix available in the HDF5-formatted restart file (also known as the “runtape” file), which is produced by the upcoming public release of the MCNP R code, version 6.3. The fission matrix, its eigenvalues, and its eigenvectors, are important tools for characterizing fissile systems such as research and power reactors and for accelerating the convergence of the Monte Carlo fission-source site distribution.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

A Python Tool for Reconstructing MCNP6 Particle Histories from an HDF5 PTRAC File [Slides]

A Python tool for converting the MCNP6 HDF5 PTRAC file to a list of Python trees is presented. The particle trees store MCNP6 simulated events for each history using parent-child relationships, which ensures that branching processes are accurately reproduced. A variety of post-processing scripts are presented and used in conjunction with the Python particle trees to make special tallies that are currently not available in the MCNP6 software and visualize the particle tracks.

97 MATHEMATICS AND COMPUTING↗