Leveraging the Cloud for HDF1 Software Testing
In this talk we will discuss how we leverage the Cloud for HDF software daily regression testing including testing of the HDF5 parallel library on the Cloud cluster using Orange FS.
CI testing↗
SEARCH · Engineering Papers
Results for “hdf5”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
LeptonInjector and LeptonWeighter: A neutrino event generator and weighter for neutrino observatories
In this work, we present a high-energy neutrino event generator, called LeptonInjector, alongside an event weighter, called LeptonWeighter. Both are designed for large-volume Cherenkov neutrino telescopes such as IceCube. The neutrino event generator allows for quick and flexible simulation of neutrino events within and around the detector volume, and implements the leading Standard Model neutrino interaction processes relevant for neutrino observatories: neutrino-nucleon deep-inelastic scattering and neutrino-electron annihilation. In this paper, we discuss the event generation algorithm, the weighting algorithm, and the main functions of the publicly available code, with examples. Program summary: Program Titles: LeptonInjector and LeptonWeighter CPC Library link to program files: https://doi.org/10.17632/662gkpjfd9.1 Developer's repository links: https://github.com/icecube/LeptonInjector and https://github.com/icecube/LeptonWeighter Licensing provisions: GNU Lesser General Public License, version 3. Programming Language: C++11 External Routines: • Boost • HDF5 • nuflux (https://github.com/icecube/nuflux) • nuSQuIDS (https://github.com/arguelles/nuSQuIDS) • Photospline (https://github.com/icecube/photospline) • SuiteSparse (https://github.com/DrTimothyAldenDavis/SuiteSparse) Nature of problem: LeptonInjector: Generate neutrino interaction events of all possible topologies and energies throughout and around a detector volume. LeptonWeighter: Reweight Monte Carlo events, generated by a set of LeptonInjector Generators, to any desired physical neutrino flux or cross section. Solution method: LeptonInjector: Projected ranges of generated leptons and the extent of the detector, in terms of column depth, are used to inject events in and around the detector volume. Event kinematics follow distributions provided in cross section files. LeptonWeighter: Event generation probabilities are calculated for each Generator, which are then combined into a generation weight and used to calculate an overall event weight.
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
py4DSTEM: A Software Package for Four-Dimensional Scanning Transmission Electron Microscopy Data Analysis
Scanning transmission electron microscopy (STEM) allows for imaging, diffraction, and spectroscopy of materials on length scales ranging from microns to atoms. By using a high-speed, direct electron detector, it is now possible to record a full two-dimensional (2D) image of the diffracted electron beam at each probe position, typically a 2D grid of probe positions. These 4D-STEM datasets are rich in information, including signatures of the local structure, orientation, deformation, electromagnetic fields, and other sample-dependent properties. However, extracting this information requires complex analysis pipelines that include data wrangling, calibration, analysis, and visualization, all while maintaining robustness against imaging distortions and artifacts. In this paper, we present py4DSTEM, an analysis toolkit for measuring material properties from 4D-STEM datasets, written in the Python language and released with an open-source license. We describe the algorithmic steps for dataset calibration and various 4D-STEM property measurements in detail and present results from several experimental datasets. We also implement a simple and universal file format appropriate for electron microscopy data in py4DSTEM, which uses the open-source HDF5 standard. We hope this tool will benefit the research community and help improve the standards for data and computational methods in electron microscopy, and we invite the community to contribute to this ongoing project.
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
mzapy : An Open-Source Python Library Enabling Efficient Extraction and Processing of Ion Mobility Spectrometry-Mass Spectrometry Data in the MZA File Format
We have recently reported MZA, a new and simple mass spectrometry data structure based on the broadly supported HDF5 format and created to facilitate software development. While this format is inherently supportive of application development, the availability of a core library with standard mass spectrometry utilities greatly facilitates fast software development. Here, we present a Python library, mzapy, for efficient extraction and processing of mass spectrometry data in the MZA format. In addition to raw data extraction, mzapy contains supporting utilities enabling tasks including calibration, signal processing, peak finding, and generating plots. Being implemented in pure Python with minimal and largely standardized dependencies makes mzapy uniquely suited to application development in the multi-omics domain. The free and open source mzapy is built with extensibility in mind, and future development will support cloud computing and artificial intelligence/machine learning applications. The software source code is freely available at https://github.com/PNNL-m-q/mzapy.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
MZA: A Data Conversion Tool to Facilitate Software Development and Artificial Intelligence Research in Multidimensional Mass Spectrometry
Modern mass spectrometry-based workflows employing hybrid instrumentation and orthogonal separations collect multidimensional data, potentially allowing deeper understanding in omics studies through adoption of artificial intelligence methods. However, the large volume of these rich data challenges existing data storage and access technologies, therefore precluding informatics advancements. Here we present MZA™ (pronounced m-za), the mass-to-charge (m/z) generic data storage and access tool designed to facilitate software development and artificial intelligence research in multidimensional mass spectrometry measurements. Composed by a data conversion tool and a simple file structure based on the HDF5 format, MZA provides easy, cross-platform and efficient programmatic access to raw MS-data, enabling fast development of new tools in data science programming languages such as Python and R. The software executable and example Python and R scripts are freely available at https://github.com/PNNL-m-q/mza.
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
A synthetic building operation dataset
Abstract This paper presents a synthetic building operation dataset which includes HVAC, lighting, miscellaneous electric loads (MELs) system operating conditions, occupant counts, environmental parameters, end-use and whole-building energy consumptions at 10-minute intervals. The data is created with 1395 annual simulations using the U.S. DOE detailed medium-sized reference office building, and 30 years’ historical weather data in three typical climates including Miami, San Francisco, and Chicago. Three energy efficiency levels of the building and systems are considered. Assumptions regarding occupant movements, occupants’ diverse temperature preferences, lighting, and MELs are adopted to reflect realistic building operations. A semantic building metadata schema - BRICK, is used to store the building metadata. The dataset is saved in a 1.2 TB of compressed HDF5 file. This dataset can be used in various applications, including building energy and load shape benchmarking, energy model calibration, evaluation of occupant and weather variability and their influences on building performance, algorithm development and testing for thermal and energy load prediction, model predictive control, policy development for reinforcement learning based building controls.
24 POWER TRANSMISSION AND DISTRIBUTION↗
Grid-based minimization at scale: Feldman-Cousins corrections for light sterile neutrino search
High Energy Physics (HEP) experiments generally employ sophisticated statistical methods to present results in searches of new physics. In the problem of searching for sterile neutrinos, likelihood ratio tests are applied to short-baseline neutrino oscillation experiments to construct confidence intervals for the parameters of interest. The test statistics of the form Δχ2 is often used to form the confidence intervals, however, this approach can lead to statistical inaccuracies due to the small signal rate in the region-of-interest. In this paper, we present a computational model for the computationally expensive Feldman-Cousins corrections to construct a statistically accurate confidence interval for neutrino oscillation analysis. The program performs a grid-based minimization over oscillation parameters and is written in C++. Our algorithms make use of vectorization through Eigen3, yielding a single-core speed-up of 350 compared to the original implementation, and achieve MPI data parallelism by employing DIY. We demonstrate the strong scaling of the application at High-Performance Computing (HPC) sites. We utilize HDF5 along with HighFive to write the results of the calculation to file.
Wospakrik, Marianette↗
Collision Tracking in OpenMC: Methods and Applications in Neutron Noise, Neutron Imaging, Time-of-Flight, and Multiplicity Counting
We present the development and application of a collision tracking feature within the OpenMC Monte Carlo particle transport code, designed for diverse applications such as neutron spectroscopy, scatter camera system, neutron noise, and multiplicity counting simulations. This feature enables the tracking of individual particle collisions, with potential applications in nuclear nonproliferation, reactor physics, and nuclear security. Additionally, the feature holds potential for the calibration of neutron detectors, specifically in converting light output into energy deposited within the detectors. The implementation consists of a set of filters—such as reaction type, energy, cell, and material—that constrain the set of collisions that are tracked, extensions to the Python API to enable simple input specification, and support for writing either OpenMC’s native HDF5-based format or the Monte Carlo particle list format. This feature was added to the official OpenMC release in version 0.15.3. In this work, the feature will be applied to showcase scenarios such as time-of-flight simulations, scatter-camera imaging for neutron source localization, neutron-noise analysis to extract integral kinetic parameters such as the prompt decay constant α, and multiplicity counting to estimate the mass of special nuclear materials. Ultimately, this feature aims to expand the application scope of open-source Monte Carlo particle transport codes such as OpenMC.
Monte Carlo code↗
Solution scattering at the Life Science X-ray Scattering (LiX) beamline
This work reports the instrumentation and software implementation at the Life Science X-ray Scattering (LiX) beamline at NSLS-II in support of biomolecular solution scattering. For automated static measurements, samples are stored in PCR tubes and grouped in 18-position sample holders. Unattended operations are enabled using a six-axis robot that exchanges sample holders between a storage box and a sample handler, transporting samples from the PCR tubes to the X-ray beam for scattering measurements. The storage box has a capacity of 20 sample holders. At full capacity, the measurements on all samples last for ∼9 h. For in-line size-exclusion chromatography, the beamline-control software coordinates with a commercial high-performance liquid chromatography (HPLC) system to measure multiple samples in batch mode. The beamline can switch between static and HPLC measurements instantaneously. In all measurements, the scattering data span a wide q -range of typically 0.006–3.2 Å −1 . Functionalities in the Python package py4xs have been developed to support automated data processing, including azimuthal averaging, merging data from multiple detectors, buffer scattering subtraction, data storage in HDF5 format and exporting the final data in a three-column text format that is acceptable by most data analysis tools. These functionalities have been integrated into graphical user interfaces that run in Jupyter notebooks, with hooks for external data analysis software.
60 APPLIED LIFE SCIENCES↗
Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL
Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.
Godoy, William↗
Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities
We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This makes it a scalability challenge, as the number of experiments stored increases in a single ensemble file. The present work follows up on our previous efforts on data management algorithms, to address identified input output (I/O) bottlenecks in Mantid, an open-source data analysis framework used across several neutron science facilities around the world. We reuse an in-memory binary-tree metadata index that resembles data access patterns, to provide a scalable search and extraction mechanism. In addition, several memory operations are refactored and optimized for the current common use cases, ranging most frequently from 10 to 180, and up to 360 separate measurement configurations. Results from this work show consistent speed ups in wall-clock time on the Mantid LoadMD routine, ranging from 19% to 23% on average, on ORNL production computing systems. The latter depends on the complexity of the targeted instrument-specific data and the system I/O and compute variability for the shared computational resources available to users of ORNL’s Spallation Neutron Source (SNS) and the High Flux Isotope Reactor (HFIR) instruments. Nevertheless, we continue to highlight the need for more research to address reduction challenges as experimental data volumes, user time and processing costs increase.
Godoy, William↗
DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics
The combination of ever-growing scientific datasets and distributed workflow complexity creates I/O performance bottlenecks due to data volume, velocity, and variety. Although the increasing use of descriptive data formats (e.g., HDF5, netCDF) helps organize these datasets, it also creates obscure bottlenecks due to the need to translate high level operations into file addresses and then into low-level I/O operations. To address this challenge, we introduce DaYu, a method and toolset for analyzing (a) semantic relationships between logical datasets and file addresses, (b) how dataset operations translate into I/O, and (c) the combination across entire workflows. DaYu's analysis and visualization enables identification of critical bottlenecks and reasoning about remediation. We describe our methodology and propose optimization guidelines. Evaluation on scientific workflows demonstrates up to 3.7x performance improvements in I/O time for obscure bottlenecks. The time and storage overhead for DaYu's time-ordered data is typically under 0.2% of runtime and 0.25% of data volume, respectively.
Tang, Meng↗
Accelerating Flash-X Simulations with Asynchronous I/O
Most high-fidelity physics simulation codes, such as Flash-X, need to save intermediate results (checkpoint files) to restart or gain insights into the evolution of the simulation. These simulation codes save such intermediate files synchronously, where computation is stalled while the data is written to storage. Depending on the problem size and computational requirements, this file write time can be a substantial portion of the total simulation time. In order to hide the I/O latency of checkpointing, asynchronous I/O methods have been introduced. These methods use background threads for performing I/O while the main threads continue with the simulation. The usage of background threads can compete for resources on the node as well as with communication. In this paper, we evaluate the overheads and the overall benefit of asynchronous I/O in HDF5 to simulations. Results from real-world high-fidelity simulations on the Summit supercomputer show that I/O operation is overlapped with application communication or computation or both, effectively hiding some or all of the I/O latency. Our evaluation shows that while using asynchronous I/O adds overhead to the application, the I/O time reduction is more significant, resulting in overall up to 1.5X performance speedup.
Jain, Rajeev↗
Towards Interactive, Reproducible Analytics at Scale on HPC Systems
The growth in scientific data volumes has resulted in a need to scale up processing and analysis pipelines using High Performance Computing (HPC) systems. These workflows need interactive, reproducible analytics at scale. The Jupyter platform provides core capabilities for interactivity but was not designed for HPC systems. In this paper, we outline our efforts that bring together core technologies based on the Jupyter Platform to create interactive, reproducible analytics at scale on HPC systems. Our work is grounded in a real world science use case-applying geophysical simulations and inversions for imaging the subsurface. Our core platform addresses three key areas of the scientific analysis workflow-reproducibility, scalability, and interactivity. We describe our implemention of a system, using Binder, Science Capsule, and Dask software. We demonstrate the use of this software to run our use case and interactively visualize real-Time streams of HDF5 data.
containers↗
AMRIC: A Novel In Situ Lossy Compression Framework for Efficient I/O in Adaptive Mesh Refinement Applications
As supercomputers advance towards exascale capabilities, computational intensity increases significantly, and the volume of data requiring storage and transmission experiences exponential growth. Adaptive Mesh Refinement (AMR) has emerged as an effective solution to address these two challenges. Concurrently, error-bounded lossy compression is recognized as one of the most efficient approaches to tackle the latter issue. Despite their respective advantages, few attempts have been made to investigate how AMR and error-bounded lossy compression can function together. To this end, this study presents a novel in-situ lossy compression framework that employs the HDF5 filter to improve both I/O costs and boost compression quality for AMR applications. We implement our solution into the AMReX framework and evaluate on two real-world AMR applications, Nyx and WarpX, on the Summit supercomputer. Experiments with 512 cores demonstrate that AMRIC improves the compression ratio by 81X and the I/O performance by 39X over AMReX's original compression solution.
Wang, Daoce↗
Evaluation of serial crystallographic structure determination within megahertz pulse trains
All "hits" in hdf5 files (Cheetah slab format) together with geometries and all CrystFEL .stream files, .hkl files and .mtz files. Also all Phenix .pdb files together with log files. All scripts needed to reproduce the result are also included.
European XFEL↗
Diffraction imaging of light induced dynamics in xenon-doped helium nanodroplets
Time-resolved wide-angle coherent diffraction imaging of individual helium nanodroplets, doped with xenon and excited with an 800 nm NIR laser pulse. Raw and preprocessed pattern, together with useful metadata, are contained in HDF5 files. Every field in the H5 container comes with a description, under the field "description", and the actual data under the field "data". For further information about the dataset see the [ArXiv article](https://arxiv.org/abs/2205.04154 ) and Bruno Langbehn's [PhD thesis](https://depositonce.tu-berlin.de/handle/11303/13189).
FERMI FEL-1↗
Diffraction imaging of light induced dynamics in xenon-doped helium nanodroplets
Time-resolved wide-angle coherent diffraction imaging of individual helium nanodroplets, doped with xenon and excited with an 800 nm NIR laser pulse. Raw and preprocessed pattern, together with useful metadata, are contained in HDF5 files. Every field in the H5 container comes with a description, under the field "description", and the actual data under the field "data". For further information about the dataset see the [ArXiv article](https://arxiv.org/abs/2205.04154 ) and Bruno Langbehn's [PhD thesis](https://depositonce.tu-berlin.de/handle/11303/13189).
FERMI FEL-1↗