Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Hierarchical Data Format”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

Counter Unmanned Aircraft System Metrics Tool

SAND2023-05387O The Counter Unmanned Aircraft System (CUAS) Metrics Tool consists of three applications: • The Data Collection Tool is designed to be used during testing and captures details that can be imported into the Figure Creator application. • The Unmanned Aircraft System (UAS) Log Converter converts different UAS logs into a standardized hierarchical data format with a timeline of geospatial data. • The Figure Creator develops relevant images for the test event. The application includes logic to ingest and correlate data from the Data Collection Tool and the UAS Log Converter to produce the chosen figures. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Mayle, Ashley↗
DOE OSTI2022

Coincident Capture through Post-processing PTRAC [Slides]

This presentation discusses the new PTRAC capabilities and workflows. The PTRAC capability in MCNP6.3 has seen a massive overhaul since MCNP6.2. The new HDF5 file format allows for both MPI- and thread-based parallelism. MCNPTools has been updated to handle the new HDF5 PTRAC format and is now open sourced on GitHub. Built-in capabilities, such as the pulse-height tally coincident capture special treatment, can largely be replicated through separate postprocessing scripts that leverage both PTRAC and MCNPTools. This allows for greater flexibility in user-specified and controlled detector response functionality, ultimately using the MCNP code for what it is best at (i.e., particle transport).

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
NASA NTRS1992

Multifrequency data analysis software on STARLINK

Although the STARLINK project was set up to provide image processing facilities to UK astronomers, it has grown over the last 12 years to the extent that it now provides most of the data analysis facilities for UK astronomers. One aspect of the growth of the STARLINK network is that it now has to cater for astronomers working in a diverse range of wavelengths. Since a given individual may be working with data obtained in a variety of wavelengths, it is most convenient if the data can be stored in a common format and the programs that analyze the data have a similar 'look and feel'. What is known as 'STARLINK software' is obtained from many sources: STARLINK funded programmers; astronomers; foreign projects such as AIPS; generally available shareware; and commercial sources when this proves cost effective. This means that the ideal situation of a completely integrated system cannot be realized in practice. Nevertheless, many of the major packages written by STARLINK application programmers and by astronomers do use a common data format, based on the Hierarchical Data System, so that interchange of data between packages designed separately from each other is simply a matter of using the same file names. For example, as astronomer might use KAPPA to read some optical spectra off a FITS tape, then use CCDPACK to debias and flat field the data (it is easy to set up an overnight batch job to do this if there is a lot of data), then use KAPPA to have a quick look at the data and then use Figaro to reduce the spectra. It is useful to divide data analysis packages into wavelength specific packages, or even instrument specific packages, and general purpose ones. Once the instrumental signature has been removed from some data, any appropriate general purpose package can be used to analyze te data. For example, the ASTERIX package deals with x-ray data reduction, but after dealing with all of the x-ray specific processing, an astronomer may well want to find the brightness of objects in a given frame. Since ASTERIX uses the standard STARLINK data format, the astronomer can use PHOTOM or DAOPHOT 2 to measure the brightness of the objects. Although DAOPHOT was written with optical astronomy in mind, it is useful for analyzing data from several wavelengths. The ability of DAOPHOT 2 to handle non-standard point spread functions can be especially useful in many areas of astronomy.

Allan, P. M.↗
DOE OSTI2025

HDF5 in the exascale era: Delivering efficient and scalable parallel I/O for exascale applications

Accurately modeling real-world systems requires scientific applications at exascale to generate massive amounts of data and manage data storage efficiently. However, parallel input and output (I/O) faces challenges due to new application workflows and the state-of-the-art memory, interconnect, and storage architectures considered in exascale designs. The storage hierarchy has expanded with node-local persistent memory, solid-state storage, and traditional disk and tape-based storage, thus requiring efficiency at each layer and much more efficient data movement among these layers. This paper discusses how the ExaHDF5 project improved the I/O performance and data management for exascale architectures by enhancing HDF5, a widely used parallel I/O library. The team developed an Asynchronous I/O Virtual Object Layer (VOL) connector that allowed overlapping I/O with computation. They also created a Cache VOL to complement asynchronous I/O by incorporating fast storage layers, such as burst buffer and node-local storage, into the parallel I/O workflow through caching and staging data. Additionally, the team enabled data aggregation and I/O at the node level by using a Subfiling Virtual File Driver (VFD). To demonstrate superior I/O performance with HDF5 at exascale, the ExaHDF5 team collaborated with several exascale applications. In this paper, we show I/O performance improvements for three applications: Cabana (a particle-based simulation library), EQSIM (a regional earthquake simulation software), and E3SM (a climate system modeling library).

Asynchronous I/Ol↗
DOE OSTI2021

Recent MCNP6 ® Code Developments and Improvements for Nuclear Engineering Applications [Slides]

The Los Alamos MCNP Monte Carlo radiation transport code has been the international gold standard for particle transport applications for over three decades. Many developments to the code have taken place with several significant new feature additions, major improvements, and enhancements to existing features. With significant institutional and programmatic investment in the code since the time of the last public release in 2018, important code development and infrastructure modernization has taken place and remains a high priority for all ongoing efforts across the code development team.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
NASA NTRS2002

MODIS Data from the GES DISC DAAC: Moderate-Resolution Imaging Spectroradiometer (MODIS)

The Goddard Earth Sciences (GES) Distributed Active Archive Center (DAAC) is responsible for the distribution of the Level 1 data, and the higher levels of all Ocean and Atmosphere products (Land products are distributed through the Land Processes (LP) DAAC DAAC, and the Snow and Ice products are distributed though the National Snow and Ice Data Center (NSIDC) DAAC). Ocean products include sea surface temperature (SST), concentrations of chlorophyll, pigment and coccolithophores, fluorescence, absorptions, and primary productivity. Atmosphere products include aerosols, atmospheric water vapor, clouds and cloud masks, and atmospheric profiles from 20 layers. While most MODIS data products are archived in the Hierarchical Data Format-Earth Observing System (HDF-EOS 2.7) format, the ocean binned products and primary productivity products (Level 4) are in the native HDF4 format. MODIS Level 1 and 2 data are of the Swath type and are packaged in files representing five minutes of Files for Level 3 and 4 are global products at daily, weekly, monthly or yearly resolutions. Apart from the ocean binned and Level 4 products, these are in Grid type, and the maps are in the Cylindrical Equidistant projection with rectangular grid. Terra viewing (scenes of approximately 2000 by 2330 km). MODIS data have several levels of maturity. Most products are released with a provisional level of maturity and only announced as validated after rigorous testing by the MODIS Science Teams. MODIS/Terra Level 1, and all MODIS/Terra 11 micron SST products are announced as validated. At the time of this publication, the MODIS Data Support Team (MDST) is working with the Ocean Science Team toward announcing the validated status of the remainder of MODIS/Terra Ocean products. MODIS/Aqua Level 1 and cloud mask products are released with provisional maturity.

Source record↗
NASA NTRS2006

Processing TES Level-1B Data

TES L1B Subsystem is a computer program that performs several functions for the Tropospheric Emission Spectrometer (TES). The term "L1B" (an abbreviation of "level 1B"), refers to data, specific to the TES, on radiometric calibrated spectral radiances and their corresponding noise equivalent spectral radiances (NESRs), plus ancillary geolocation, quality, and engineering data. The functions performed by TES L1B Subsystem include shear analysis, monitoring of signal levels, detection of ice build-up, and phase correction and radiometric and spectral calibration of TES target data. Also, the program computes NESRs for target spectra, writes scientific TES level-1B data to hierarchical- data-format (HDF) files for public distribution, computes brightness temperatures, and quantifies interpixel signal variability for the purpose of first-order cloud and heterogeneous land screening by the level-2 software summarized in the immediately following article. This program uses an in-house-developed algorithm, called "NUSRT," to correct instrument line-shape factors.

DeBaca, Richard C.↗
NASA NTRS1992

An object-oriented data reduction system in Fortran

A data reduction system for the AAO two-degree field project is being developed using an object-oriented approach. Rather than use an object-oriented language (such as C++) the system is written in Fortran and makes extensive use of existing subroutine libraries provided by the UK Starlink project. Objects are created using the extensible N-dimensional Data Format (NDF) which itself is based on the Hierarchical Data System (HDS). The software consists of a class library, with each class corresponding to a Fortran subroutine with a standard calling sequence. The methods of the classes provide operations on NDF objects at a similar level of functionality to the applications of conventional data reduction systems. However, because they are provided as callable subroutines, they can be used as building blocks for more specialist applications. The class library is not dependent on a particular software environment thought it can be used effectively in ADAM applications. It can also be used from standalone Fortran programs. It is intended to develop a graphical user interface for use with the class library to form the 2dF data reduction system.

Bailey, J.↗
DOE OSTI2026

Enabling event-by-event precision in γ-ray cascades for neutron-induced reactions

Neutron-induced γ-ray spectra provide key inputs for modern active interrogation applications. A precise modeling of the nuclear reaction and subsequent emission of γ rays is challenging and often impossible due to limitations on evaluated data file formats and nuclear transport simulation codes. We present a framework that addresses these challenges by combining experimental data and reaction-model calculation outputs into an extended candidate version of the Generalized Nuclear Data Structure (GNDS) file, the successor format for the legacy Evaluated Nuclear Data File (ENDF-6). This proposed GNDS hierarchical format contains all the necessary ingredients for inline γ-ray cascade reproduction with event-by-event precision, including continuum–continuum and continuum–discrete transitions following neutron-capture and inelastic neutron scattering reactions. Cascade-event generation based on our approach demonstrates improved energy conservation on an event-by-event basis and permits the use of γ-γ coincidences in applications. This work offers, for the first time, a method to generate neutron-capture and inelastic neutron-scattering γ-ray cascades where energy conservation, correlations, and experimental primaries are fully accounted for.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2023

Accelerating Random Forest Classification on GPU and FPGA

Random Forests (RFs) are a commonly used machine learning method for classification and regression tasks spanning a variety of application domains, including bioinformatics, business analytics, and software optimization. While prior work has focused primarily on improving performance of the training of RFs, many applications, such as malware identification, cancer prediction, and banking fraud detection, require fast RF classification. In this work, we accelerate RF classification on GPU and FPGA. In order to provide efficient support for large datasets, we propose a hierarchical memory layout suitable to the GPU/FPGA memory hierarchy. We design three RF classification code variants based on that layout, and we investigate GPU- and FPGA-specific considerations for these kernels. Our experimental evaluation, performed on an Nvidia Xp GPU and on a Xilinx Alveo U250 FPGA accelerator card using publicly available datasets on the scale of millions of samples and tens of features, covers various aspects. First, we evaluate the performance benefits of our hierarchical data structure over the standard compressed sparse row (CSR) format. Second, we compare our GPU implementation with cuML, a machine learning library targeting Nvidia GPUs. Third, we explore the performance/accuracy tradeoff resulting from the use of different tree depths in the RF. Finally, we perform a comparative performance analysis of our GPU and FPGA implementations. Our evaluation shows that for high accuracy targets, our GPU implementation yields 5-9x speedup over CSR, and up to a 2x speedup over cuML.

FPGA, Xilinx FPGA, GPU, Random Forest classificati↗
DOE OSTI2026

TRACE Input Modernization

This work presents a Tom’s Obvious Minimal Language (TOML)-based representation of input for the US Nuclear Regulatory Commission’s TRAC/RELAP Advanced Computational Engine (TRACE) thermal hydraulics code. Implemented using the Workbench Analysis Sequence Processor (WASP), the approach maps traditional TRACE input structures to a hierarchical format composed of named parameters, typed values, and native data collections. The resulting representation preserves TRACE’s existing modeling capabilities while providing a modern, structured interface for model development and management. WASP further extends TOML through a file import directive that supports modular model composition and reusable input organization. In addition, WASP provides extended array data entry convenience with various data repeat and interpolation capabilities. Examples of the new TOML syntax are provided for major TRACE input categories, including hydraulic components, heat structures, control systems, and trip logic. The TOML representation establishes a foundation for improved validation, tooling, automation, and model maintainability while remaining compatible with existing TRACE workflows. To facilitate migration to the TOML-based input format, the TRACE executable now supports conversion of native TRACE input into an intermediate JSON representation. A Python utility subsequently transforms the JSON data into an equivalent TOML model. Lastly, the TRACE executable now supports execution using TOML-formatted input.

Lefebvre, Robert A. [Oak Ridge National Laboratory↗
NASA NTRS1994

Compression of regions in the global advanced very high resolution radiometer 1-km data set

The global advanced very high resolution radiometer (AVHRR) 1-km data set is a 10-band image produced at USGS' EROS Data Center for the study of the world's land surfaces. The image contains masked regions for non-land areas which are identical in each band but vary between data sets. They comprise over 75 percent of this 9.7 gigabyte image. The mask is compressed once and stored separately from the land data which is compressed for each of the 10 bands. The mask is stored in a hierarchical format for multi-resolution decompression of geographic subwindows of the image. The land for each band is compressed by modifying a method that ignores fill values. This multi-spectral region compression efficiently compresses the region data and precludes fill values from interfering with land compression statistics. Results show that the masked regions in a one-byte test image (6.5 Gigabytes) compress to 0.2 percent of the 557,756,146 bytes they occupy in the original image, resulting in a compression ratio of 89.9 percent for the entire image.

Kess, Barbara L.↗
NASA NTRS1989

A system architecture for a planetary rover

Each planetary mission requires a complex space vehicle which integrates several functions to accomplish the mission and science objectives. A Mars Rover is one of these vehicles, and extends the normal spacecraft functionality with two additional functions: surface mobility and sample acquisition. All functions are assembled into a hierarchical and structured format to understand the complexities of interactions between functions during different mission times. It can graphically show data flow between functions, and most importantly, the necessary control flow to avoid unambiguous results. Diagrams are presented organizing the functions into a structured, block format where each block represents a major function at the system level. As such, there are six blocks representing telecomm, power, thermal, science, mobility and sampling under a supervisory block called Data Management/Executive. Each block is a simple collection of state machines arranged into a hierarchical order very close to the NASREM model for Telerobotics. Each layer within a block represents a level of control for a set of state machines that do the three primary interface functions: command, telemetry, and fault protection. This latter function is expanded to include automatic reactions to the environment as well as internal faults. Lastly, diagrams are presented that trace the system operations involved in moving from site to site after site selection. The diagrams clearly illustrate both the data and control flows. They also illustrate inter-block data transfers and a hierarchical approach to fault protection. This systems architecture can be used to determine functional requirements, interface specifications and be used as a mechanism for grouping subsystems (i.e., collecting groups of machines, or blocks consistent with good and testable implementations).

Smith, D. B.↗
NASA NTRS1999

Earth Science Data Archive and Access at the NASA/Goddard Space Flight Center Distributed Active Archive Center (DAAC)

The Goddard Distributed Active Archive Center (DAAC), as an integral part of the Earth Observing System Data and Information System (EOSDIS), is the official source of data for several important earth remote sensing missions. These include the Sea-viewing Wide-Field-of-view Sensor (SeaWiFS) launched in August 1997, the Tropical Rainfall Measuring Mission (TRMM) launched in November 1997, and the Moderate Resolution Imaging Spectroradiometer (MODIS) scheduled for launch in mid 1999 as part of the EOS AM-1 instrumentation package. The data generated from these missions supports a host of users in the hydrological, land biosphere and oceanographic research and applications communities. The volume and nature of the data present unique challenges to an Earth science data archive and distribution system such as the DAAC. The DAAC system receives, archives and distributes a large number of standard data products on a daily basis, including data files that have been reprocessed with updated calibration data or improved analytical algorithms. A World Wide Web interface is provided allowing interactive data selection and automatic data subscriptions as distribution options. The DAAC also creates customized and value-added data products, which allow additional user flexibility and reduced data volume. Another significant part of our overall mission is to provide ancillary data support services and archive support for worldwide field campaigns designed to validate the results from the various satellite-derived measurements. In addition to direct data services, accompanying documentation, WWW links to related resources, support for EOSDIS data formats, and informed response to inquiries are routinely provided to users. The current GDAAC WWW search and order system is being restructured to provide users with a simplified, hierarchical access to data. Data Browsers have been developed for several data sets to aid users in ordering data. These Browsers allow users to specify spatial, temporal, and other parameter criteria in searching for and previewing data.

Leptoukh, Gregory↗
NASA NTRS2003

The Ionization History of The Intergalactic Medium:

The funded project seeked a unified description of the ionization, physical structure, and evolution of the intergalactic medium (IGM) and quasar intervening absorption systems. We proposed to conduct theoretical studies of the IGM and QSO absorbers in the context of current theories of galaxy formation, developing and using numerical and analytical techniques aimed at a detailed modeling of cosmological radiative transfer, gas dynamics, and thermal and ionization evolution. The ionization history of the IGM has important implications for the metagalactic UV background, intergalactic helium absorption 21-cm tomography, metal absorption systems, fluctuations in the microwave background, and the cosmic rate of structure and star formation. All the original objectives of our program have been achieved, and the results widely used and quoted by the community. Indeed, they remain relevant as the level and complexity of research in this area has increased substantially since our proposal was submitted, due to new discoveries on galaxy formation and evolution, a flood of high-quality data on the distant universe, new theoretical ideas and direct numerical simulations of structure formation in hierarchical clustering theories.

Madau, Piero↗
NASA NTRS2000

Formation Flying With Decentralized Control in Libration Point Orbits

A decentralized control framework is investigated for applicability of formation flying control in libration orbits. The decentralized approach, being non-hierarchical, processes only direct measurement data, in parallel with the other spacecraft. Control is accomplished via linearization about a reference libration orbit with standard control using a Linear Quadratic Regulator (LQR) or the GSFC control algorithm. Both are linearized about the current state estimate as with the extended Kalman filter. Based on this preliminary work, the decentralized approach appears to be feasible for upcoming libration missions using distributed spacecraft.

Folta, David↗