Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “hdf5”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

Collision Tracking in OpenMC: Methods and Applications in Neutron Noise, Neutron Imaging, Time-of-Flight, and Multiplicity Counting

We present the development and application of a collision tracking feature within the OpenMC Monte Carlo particle transport code, designed for diverse applications such as neutron spectroscopy, scatter camera system, neutron noise, and multiplicity counting simulations. This feature enables the tracking of individual particle collisions, with potential applications in nuclear nonproliferation, reactor physics, and nuclear security. Additionally, the feature holds potential for the calibration of neutron detectors, specifically in converting light output into energy deposited within the detectors. The implementation consists of a set of filters—such as reaction type, energy, cell, and material—that constrain the set of collisions that are tracked, extensions to the Python API to enable simple input specification, and support for writing either OpenMC’s native HDF5-based format or the Monte Carlo particle list format. This feature was added to the official OpenMC release in version 0.15.3. In this work, the feature will be applied to showcase scenarios such as time-of-flight simulations, scatter-camera imaging for neutron source localization, neutron-noise analysis to extract integral kinetic parameters such as the prompt decay constant α, and multiplicity counting to estimate the mass of special nuclear materials. Ultimately, this feature aims to expand the application scope of open-source Monte Carlo particle transport codes such as OpenMC.

Monte Carlo code↗
DOE OSTI2020

Solution scattering at the Life Science X-ray Scattering (LiX) beamline

This work reports the instrumentation and software implementation at the Life Science X-ray Scattering (LiX) beamline at NSLS-II in support of biomolecular solution scattering. For automated static measurements, samples are stored in PCR tubes and grouped in 18-position sample holders. Unattended operations are enabled using a six-axis robot that exchanges sample holders between a storage box and a sample handler, transporting samples from the PCR tubes to the X-ray beam for scattering measurements. The storage box has a capacity of 20 sample holders. At full capacity, the measurements on all samples last for ∼9 h. For in-line size-exclusion chromatography, the beamline-control software coordinates with a commercial high-performance liquid chromatography (HPLC) system to measure multiple samples in batch mode. The beamline can switch between static and HPLC measurements instantaneously. In all measurements, the scattering data span a wide q -range of typically 0.006–3.2 Å −1 . Functionalities in the Python package py4xs have been developed to support automated data processing, including azimuthal averaging, merging data from multiple detectors, buffer scattering subtraction, data storage in HDF5 format and exporting the final data in a three-column text format that is acceptable by most data analysis tools. These functionalities have been integrated into graphical user interfaces that run in Jupyter notebooks, with hooks for external data analysis software.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2020

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXus data access patterns to provide a scalable search and extraction mechanism. Second, data encapsulation in Mantid algorithms is optimally redesigned to reduce the total compute and memory runtime footprint associated with metadata I/O reconstruction tasks. Results from this work show speed ups in wall-clock time on ORNL data reduction workflows, ranging from 11% to 30% depending on the complexity of the targeted instrument-specific data. Nevertheless, we highlight the need for more research to address reduction challenges as experimental data volumes increase.

Godoy, William↗
DOE OSTI2021

Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities

We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This makes it a scalability challenge, as the number of experiments stored increases in a single ensemble file. The present work follows up on our previous efforts on data management algorithms, to address identified input output (I/O) bottlenecks in Mantid, an open-source data analysis framework used across several neutron science facilities around the world. We reuse an in-memory binary-tree metadata index that resembles data access patterns, to provide a scalable search and extraction mechanism. In addition, several memory operations are refactored and optimized for the current common use cases, ranging most frequently from 10 to 180, and up to 360 separate measurement configurations. Results from this work show consistent speed ups in wall-clock time on the Mantid LoadMD routine, ranging from 19% to 23% on average, on ORNL production computing systems. The latter depends on the complexity of the targeted instrument-specific data and the system I/O and compute variability for the shared computational resources available to users of ORNL’s Spallation Neutron Source (SNS) and the High Flux Isotope Reactor (HFIR) instruments. Nevertheless, we continue to highlight the need for more research to address reduction challenges as experimental data volumes, user time and processing costs increase.

Godoy, William↗
DOE OSTI2024

DaYu: Optimizing Distributed Scientific Workflows by Decoding Dataflow Semantics and Dynamics

The combination of ever-growing scientific datasets and distributed workflow complexity creates I/O performance bottlenecks due to data volume, velocity, and variety. Although the increasing use of descriptive data formats (e.g., HDF5, netCDF) helps organize these datasets, it also creates obscure bottlenecks due to the need to translate high level operations into file addresses and then into low-level I/O operations. To address this challenge, we introduce DaYu, a method and toolset for analyzing (a) semantic relationships between logical datasets and file addresses, (b) how dataset operations translate into I/O, and (c) the combination across entire workflows. DaYu's analysis and visualization enables identification of critical bottlenecks and reasoning about remediation. We describe our methodology and propose optimization guidelines. Evaluation on scientific workflows demonstrates up to 3.7x performance improvements in I/O time for obscure bottlenecks. The time and storage overhead for DaYu's time-ordered data is typically under 0.2% of runtime and 0.25% of data volume, respectively.

Tang, Meng↗
DOE OSTI2022

Accelerating Flash-X Simulations with Asynchronous I/O

Most high-fidelity physics simulation codes, such as Flash-X, need to save intermediate results (checkpoint files) to restart or gain insights into the evolution of the simulation. These simulation codes save such intermediate files synchronously, where computation is stalled while the data is written to storage. Depending on the problem size and computational requirements, this file write time can be a substantial portion of the total simulation time. In order to hide the I/O latency of checkpointing, asynchronous I/O methods have been introduced. These methods use background threads for performing I/O while the main threads continue with the simulation. The usage of background threads can compete for resources on the node as well as with communication. In this paper, we evaluate the overheads and the overall benefit of asynchronous I/O in HDF5 to simulations. Results from real-world high-fidelity simulations on the Summit supercomputer show that I/O operation is overlapped with application communication or computation or both, effectively hiding some or all of the I/O latency. Our evaluation shows that while using asynchronous I/O adds overhead to the application, the I/O time reduction is more significant, resulting in overall up to 1.5X performance speedup.

Jain, Rajeev↗
DOE OSTI2020

Towards Interactive, Reproducible Analytics at Scale on HPC Systems

The growth in scientific data volumes has resulted in a need to scale up processing and analysis pipelines using High Performance Computing (HPC) systems. These workflows need interactive, reproducible analytics at scale. The Jupyter platform provides core capabilities for interactivity but was not designed for HPC systems. In this paper, we outline our efforts that bring together core technologies based on the Jupyter Platform to create interactive, reproducible analytics at scale on HPC systems. Our work is grounded in a real world science use case-applying geophysical simulations and inversions for imaging the subsurface. Our core platform addresses three key areas of the scientific analysis workflow-reproducibility, scalability, and interactivity. We describe our implemention of a system, using Binder, Science Capsule, and Dask software. We demonstrate the use of this software to run our use case and interactively visualize real-Time streams of HDF5 data.

containers↗
DOE OSTI2023

AMRIC: A Novel In Situ Lossy Compression Framework for Efficient I/O in Adaptive Mesh Refinement Applications

As supercomputers advance towards exascale capabilities, computational intensity increases significantly, and the volume of data requiring storage and transmission experiences exponential growth. Adaptive Mesh Refinement (AMR) has emerged as an effective solution to address these two challenges. Concurrently, error-bounded lossy compression is recognized as one of the most efficient approaches to tackle the latter issue. Despite their respective advantages, few attempts have been made to investigate how AMR and error-bounded lossy compression can function together. To this end, this study presents a novel in-situ lossy compression framework that employs the HDF5 filter to improve both I/O costs and boost compression quality for AMR applications. We implement our solution into the AMReX framework and evaluate on two real-world AMR applications, Nyx and WarpX, on the Summit supercomputer. Experiments with 512 cores demonstrate that AMRIC improves the compression ratio by 81X and the I/O performance by 39X over AMReX's original compression solution.

Wang, Daoce↗
DOE OSTI2022

Diffraction imaging of light induced dynamics in xenon-doped helium nanodroplets

Time-resolved wide-angle coherent diffraction imaging of individual helium nanodroplets, doped with xenon and excited with an 800 nm NIR laser pulse. Raw and preprocessed pattern, together with useful metadata, are contained in HDF5 files. Every field in the H5 container comes with a description, under the field "description", and the actual data under the field "data". For further information about the dataset see the [ArXiv article](https://arxiv.org/abs/2205.04154 ) and Bruno Langbehn's [PhD thesis](https://depositonce.tu-berlin.de/handle/11303/13189).

FERMI FEL-1↗
DOE OSTI2022

Diffraction imaging of light induced dynamics in xenon-doped helium nanodroplets

Time-resolved wide-angle coherent diffraction imaging of individual helium nanodroplets, doped with xenon and excited with an 800 nm NIR laser pulse. Raw and preprocessed pattern, together with useful metadata, are contained in HDF5 files. Every field in the H5 container comes with a description, under the field "description", and the actual data under the field "data". For further information about the dataset see the [ArXiv article](https://arxiv.org/abs/2205.04154 ) and Bruno Langbehn's [PhD thesis](https://depositonce.tu-berlin.de/handle/11303/13189).

FERMI FEL-1↗
DOE OSTI2020

Quick n Dirty

QnD is a pure Python package for working with binary files. It has built-in support for HDF5, NetCDF3, and PDB formats. It provides a Pythonic, dict-like interface, and allows easy movement among time-like "records". Both reading and writing are enabled. The HDF and NetCDF capabilities rely on external libraries, while the PDB support is part of QnD itself and entirely in Python. QnD works under Python 2 and 3. For interactive use, much less typing is involved than for some other packages.

MUNRO, DavidH.↗
DOE OSTI2020

PFLOTRAN

PFLOTRAN is a GNU LGPL licensed code that simulates single phase water and multiphase (air, co2, water) flow, energy transport, geochemical reaction and biological reaction in the earth's subsurface environment. The problem may be discretized on structured and unstructured grids. The code is designed to run in parallel on supercomputers and leverages the open source PETSc library for solvers and data structures and open source HDF5 for binary file I/O. See www.pflotran.org and/or doc-dev.pflotran.org for further information. SAND2020-12161 M Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Hammond, Glenn↗
DOE OSTI2021

HangulFonts v1.0.0

The software creates the Hangul Fonts data set.The Hangul Fonts Datasets consists of all possible blocks for each font along with a set of labels and features for each block which are common across all fonts. We provide scripts to render each block and font into an image and store them together in an HDF5 file for each font and fontsize specified. Each font has 11,172 possible blocks and we have curated 35 fonts for a total of 391,020 possible images. The spoken consonants and vowel correspond to a set of "initial", "medial", and "final" labels for each block with 19, 21, and 28 classes respectively. In addition, each block can also be assigned a hierarchy label based on the block geometry. Finally, the individual glyphs (atoms) are composed across the initial, medial, and final locations along with potential scalings, translations, or rotations. These features of the data set make it uniquely positioned in the deep representation learning field because of its compositionally and hierarchical organization, while retaining diverse types of variation in the realizations.

Bouchard, Kristofer↗
DOE OSTI2022

Microscopy Parser

Reads proprietary electron microscope vendor files and exports them in a standardized open source HDF5 format.

Simpson, Joseph↗
DOE OSTI2024

Data-Driven Protection Software to classify fault locations by protective zone in distribution systems with high PV penetration

The software contains (a) the source codes to generate Point-on-Wave (PoW) transient data for any feeder model in Alternative Transient Program (ATP) format. Codes provide options to change different steady state settings, including the loading condition and PV capacity and transient state setting like faults type, location and initiation time (b) data post-processing source code to converted data from native format to COMTRADE, csv, HDF5 (c) Docker container to train CNN to classify fault locations by protective zone. The container takes dataset and other training parameters (sampling rate, training epochs, batch size etc) as input to train CNN. The container writes back the trained CNN model, training and testing metrics and plots to the local workstation

Ramesh, Meghana↗
DOE OSTI2021

OpenMC

OpenMC is an open source, community-developed framework for performing Monte Carlo particle transport calculations.It is capable of simulating neutron and photon transport below ~50 MeV and also includes a model for bremsstrahlung production of photons by electrons/positrons. Fixed source, k-eigenvalue, and subcritical multiplication problems can be solved. Problem geometries can be modeled either using constructive solid geometry or a CAD representation. A flexible and efficient tally system enables a wide variety of physical quantities to be tallied and analyzed. OpenMC can run in parallel using a hybrid MPI and OpenMP programming model and has been extensively tested on leadership class supercomputers. A built-in Bateman equation solver enables the modeling of material composition changes due to irradiation. One of the unique features of OpenMC is its rich, extensible Python and C/C++ programming interfaces that enable programming pre- and post-processing, multigroup cross section generation, workflow automation, multiphysics coupling, and the visualization of geometry and tally results. In addition to the core Monte Carlo transport solver and associated APIs, OpenMC includes a Python-based nuclear data interface that enables power users to inspect, modify,and perform various types of nuclear data processing on ENDF, ACE, and OpenMC¿s native HDF5 files

ROMANO, PaulK.↗
DOE OSTI2025

Dirty Word Scanner

SAND2025-09142O Dirty Word Scanner helps prevent the accidental inclusion of sensitive terms by scaning files in repositories to catch "dirty words" before they are committed. While there are existing solutions focused on passwords and API keys, this tool offers additional features tailored to specific security needs. It will function as a standalone tool, incorporating advanced capabilities from similar tools to provide a comprehensive solution. This tool can unpack HDF5 files and examine their contents. It can display image, audio, and visual files to the user and request a manual determination of whether they are safe. It can also detect arbitrary binary files and ask the user to verify that they're safe. The tool enables sophisticated whitelisting of strings and regular expressions for cases where a term is sensitive in certain contexts but not in others. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Gates, Jason [Sandia National Lab. (SNL-CA), Liver↗