Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “hierarchical data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS2000

Subsetting and Formatting Landsat-7 LOR ETM+ and Data Products

The Landsat-7 Processing System (LPS) processes Landsat-7 Enhanced Thematic Mapper (ETM+) instrument data into large, contiguous segments called "subintervals" and stores them in Level OR (LOR) data files. The LPS processed subinterval products must be subsetted and reformatted before the Level I processing systems can ingest them. The initial full subintervals produced by the LPS are stored mainly in HDF Earth Observing System (HDF-EOS) format which is an extension to the Hierarchical Data Format (HDF). The final LOR products are stored in native HDF format. Primarily the EOS Core System (ECS) and alternately the DAAC Emergency System (DES) subset the subinterval data for the operational Landsat-7 data processing systems. The HDF and HDF-EOS application programming interfaces (APIs) can be used for extensive data subsetting and data reorganization. A stand-alone subsetter tool has been developed which is based on some of the DES code. This tool makes use of the HDF and HDFEOS APIs to perform Landsat-7 LOR product subsetting and demonstrates how HDF and HDFEOS can be used for creating various configurations of full LOR products. How these APIs can be used to efficiently subset, format, and organize Landsat-7 LOR data as demonstrated by the subsetter tool and the DES is discussed.

Reid, Michael R.↗
NASA NTRS2001

Evaluation of Incremental Releases of ECS User Interfaces and the Development of HDF/HDF-EOS Tutorials

During the reporting period, the PI has continued to serve on numerous review panels, task forces, and committees with the goal of providing input and guidance for the Earth Observing System Data and Information System (EOSDIS) program at NASA Headquarters and NASA Goddard Space Flight Center (GSFC). In addition, the PI has worked together with personnel at Simpson Weather Associates (SWA) to help create an on-line HDF/HDF-EOS tutorial for beginning and non-expert users of both the Hierarchical Data Format (HDF) and HDF-EOS data format and software libraries. Finally, the PI has worked together with personnel at SWA and the Information Technology and Systems Center (ITSC) at the University of Alabama in Huntsville (UAH) on a feasibility study regarding the use of data mining software to ascertain features from the gridded output from numerical meteorological forecast models. A summary of these activities is provided.

Emmitt, G. D.↗
DOE OSTI2026

Model Data Archive Associated with Manuscript "Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon"

This data package supports the publication “Fire-altered Carbon Pools Create Disturbance Memory in Stream Dissolved Organic Carbon” by Li et al. (2026). The package contains processed model inputs, configuration files, restart files, simulation outputs, scripts, and visualization products used to evaluate post-fire dissolved organic carbon (DOC) dynamics in the Naches River Watershed, Washington, USA, following the 2021 Schneider Springs Fire. The modeling workflow couples ELM-BGC, the biogeochemistry-enabled Energy Exascale Earth System Model Land Model; ATS, the Advanced Terrestrial Simulator for integrated surface-subsurface hydrology; and PFLOTRAN, a reactive transport model for multicomponent aqueous geochemistry. Together, these models simulate how wildfire-induced changes in vegetation, litter, coarse woody debris, and soil organic matter influence DOC production, transport, and reaction from burned hillslopes to stream networks. The archive includes preprocessed meteorological, geospatial, hydrologic, and biogeochemical forcing data; ELM-BGC-derived DOC source terms; ATS mesh files; PFLOTRAN reactive-transport inputs; model configuration files; spin-up and transient restart files; watershed-scale diagnostic outputs; stream concentration time series; and figures or visualization files used to inspect and reproduce key results. File types include Hierarchical Data Format 5 (HDF5) files for gridded forcing and model-coupling data, model input and configuration files for ELM-BGC, ATS, and PFLOTRAN, restart and simulation-output files generated by the modeling workflow, tabular or time-series diagnostic outputs, scripts for post-processing and figure generation, and image or visualization products associated with the manuscript. Use of the package depends on the intended task. Re-running the simulations requires the relevant modeling software, including ELM-BGC, ATS, and PFLOTRAN as ATS's geochemical engine. Inspecting outputs and reproducing figures requires Python with scientific plotting libraries such as Matplotlib, and three-dimensional model outputs may be viewed with ParaView. Geographic information system files or maps may be inspected with ArcGIS Pro or comparable GIS software. The data package is intended to enable traceability, reuse, and partial reproduction of the coupled land-to-watershed hydro-biogeochemical modeling workflow used to test how wildfire disturbance affects terrestrial carbon pools and downstream DOC dynamics.

ATS↗
NASA NTRS2000

Accelerating Time-Varying Hardware Volume Rendering Using TSP Trees and Color-Based Error Metrics

This paper describes a new hardware volume rendering algorithm for time-varying data. The algorithm uses the Time-Space Partitioning (TSP) tree data structure to identify regions within the data that have spatial or temporal coherence. By using this coherence, the rendering algorithm can improve performance when the volume data is larger than the texture memory capacity by decreasing the amount of textures required. This coherence can also allow improved speed by appropriately rendering flat-shaded polygons instead of textured polygons, and by not rendering transparent regions. To reduce the polygonization overhead caused by the use of the hierarchical data structure, we introduce an optimization method using polygon templates. The paper also introduces new color-based error metrics, which more accurately identify coherent regions compared to the earlier scalar-based metrics. By showing experimental results from runs using different data sets and error metrics, we demonstrate that the new methods give substantial improvements in volume rendering performance.

Ellsworth, David↗
NASA NTRS1992

Integrated modeling and design of advanced optical systems

A new modeling, analysis and design environment for precision optical spacecraft being developed at JPL is described in terms of its language, implementation and data organization. The computer program IMOS (Integrated Modeling of Optical Systems) will provide an integrated end-to-end modeling and design environment for spacecraft missions requiring multidisciplinary system performance evaluation. IMOS discipline modules will be coded in an Ada-like language to reduce model development and evaluation time, and new hierarchical data base concepts will be included to facilitate system trade studies and tracking of system requirements. The development will support a growing community of engineers involved in integrated modeling of current and future NASA missions.

Briggs, Hugh C.↗
NASA NTRSDate not supplied

ASDC’s Python-Based Metadata Extraction Pipeline for Suborbital Campaigns

The FAIRness of data products, especially findability and accessibility depend on rich metadata which, when extracted, can allow for proper curation. Over the past few years, the Atmospheric Science Data Center (ASDC) suborbital science support team has developed a metadata extraction pipeline to ensure the required metadata can be retrieved systematically, effectively, and efficiently to ensure the data can be used by a broad community. The development of a pipeline has presented many, but necessary, challenges to support archival and distribution of ASDC’s 30+ suborbital missions. Though sufficient metadata is provided by instrument scientists, the metadata may not be readily machine actionable due to different formats and templates. Further complicating metadata extraction, our team has found that the nature of metadata can be quite diverse given the difference in measurement types, instruments, and measurement platforms. A metadata extraction pipeline has been developed to provide an efficient, plugin-in based, method for adding new parsers, a configuration system that lets non-developers customize how files are processed, and a system for identifying and logging metadata quality issues to ensure they are readily found and addressed. The metadata extraction pipeline identifies critical pieces of metadata that are needed to promote data FAIRness, including location, file revision, measurement start/end datetime and can be easily modified to extract further information (such as variables). Given the wide-ranging datasets, the pipeline has been modified to accommodate multiple file formats, including multiple versions of ICARTT (International Consortium for Atmospheric Research on Transport and Transformation), HDF (Hierarchical Data Format), netCDF (network Common Data Form), and multiple versions of the Ames File Format. The pipeline also supports building metadata for file formats that cannot have metadata easily extracted from them, such as PDF (Portable Document Format) and GIF (Graphics Interchange Format). The pipeline has allowed our team to maintain a consistent flow of data and metadata to archival and distribution services, ensuring the ASDC meets the needs of the suborbital science community. This presentation will highlight the ASDC’s suborbital metadata extraction pipeline, its development, how it’s been modified to support data FAIRness, and plans for maintaining the pipeline and adding new features.

Abraham Porter↗
DOE OSTI2024

h5bench: A unified benchmark suite for evaluating HDF5 I/O performance on pre‐exascale platforms

Summary Parallel I/O is a critical technique for moving data between compute and storage subsystems of supercomputers. With massive amounts of data produced or consumed by compute nodes, high‐performant parallel I/O is essential. I/O benchmarks play an important role in this process; however, there is a scarcity of I/O benchmarks representative of current workloads on HPC systems. Toward creating representative I/O kernels from real‐world applications, we have created h5bench , a set of I/O kernels that exercise hierarchical data format version 5 (HDF5) I/O on parallel file systems in numerous dimensions. Our focus on HDF5 is due to the parallel I/O library's heavy usage in various scientific applications running on supercomputing systems. The various tests benchmarked in the h5bench suite include I/O operations (read and write), data locality (arrays of basic data types and arrays of structures), array dimensionality (one‐dimensional arrays, two‐dimensional meshes, three‐dimensional cubes), I/O modes (synchronous and asynchronous). In this paper, we present the observed performance of h5bench executed along several of these dimensions on existing supercomputers (Cori and Summit) and pre‐exascale platforms (Perlmutter, Theta, and Polaris). h5bench measurements can be used to identify performance bottlenecks and their root causes and evaluate I/O optimizations. As the I/O patterns of h5bench are diverse and capture the I/O behaviors of various HPC applications, this study will be helpful to the broader supercomputing and I/O community.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2004

Fossil Signatures Using Elemental Abundance Distributions and Bayesian Probabilistic Classification

Elemental abundances (C6, N7, O8, Na11, Mg12, Al3, P15, S16, Cl17, K19, Ca20, Ti22, Mn25, Fe26, and Ni28) were obtained for a set of terrestrial fossils and the rock matrix surrounding them. Principal Component Analysis extracted five factors accounting for the 92.5% of the data variance, i.e. information content, of the elemental abundance data. Hierarchical Cluster Analysis provided unsupervised sample classification distinguishing fossil from matrix samples on the basis of either raw abundances or PCA input that agreed strongly with visual classification. A stochastic, non-linear Artificial Neural Network produced a Bayesian probability of correct sample classification. The results provide a quantitative probabilistic methodology for discriminating terrestrial fossils from the surrounding rock matrix using chemical information. To demonstrate the applicability of these techniques to the assessment of meteoritic samples or in situ extraterrestrial exploration, we present preliminary data on samples of the Orgueil meteorite. In both systems an elemental signature produces target classification decisions remarkably consistent with morphological classification by a human expert using only structural (visual) information. We discuss the possibility of implementing a complexity analysis metric capable of automating certain image analysis and pattern recognition abilities of the human eye using low magnification optical microscopy images and discuss the extension of this technique across multiple scales.

Hoover, Richard B.↗
DOE OSTI2021

Data shuffling with hierarchical tuple spaces

Methods and systems for shuffling data to generate a dataset are described. A first map module may generate first pair data, and a second map module may generate second pair data, from source data. The first map module may insert the first pair data into a first local tuple space accessible to the first map module. The second map module may insert the second pair data into a second local tuple space accessible to the second map module. A shuffle module may request pair data that includes a particular key. The first and second pair data may be inserted into a global tuple space accessible by the first and second map modules. The shuffle module may identify the requested pair data in the global tuple space, and may fetch the identified pair data from a memory. The shuffle module may shuffle the fetched pair data to generate the dataset.

Kayi, Abdullah↗
DOE OSTI2021

Data shuffling with hierarchical tuple spaces

Methods and systems for shuffling data are described. A processor may generate pair data from source data. The processor may insert the pair data into local tuple spaces. In response to a request for a particular key, the processor may determine a presence of the requested key in a global tuple space. The processor may, in response to a presence of the requested key in the global tuple space, update the global tuple space. The update may be based on the pair data among the local tuple spaces including the existing key. The processor may, in response to an absence of the requested key in the global tuple space, insert pair data including the missing key from the local tuple spaces into the global tuple space. The processor may fetch the requested pair data, and may shuffle the fetched data to generate a dataset.

Andrade Costa, Carlos Henrique↗
NASA NTRS2016

EVA Wiki - Transforming Knowledge Management for EVA Flight Controllers and Instructors

The EVA Wiki was recently implemented as the primary knowledge database to retain critical knowledge and skills in the EVA Operations group at NASA's Johnson Space Center by ensuring that information is recorded in a common, easy to search repository. Prior to the EVA Wiki, information required for EVA flight controllers and instructors was scattered across different sources, including multiple file share directories, SharePoint, individual computers, and paper archives. Many documents were outdated, and data was often difficult to find and distribute. In 2011, a team recognized that these knowledge management problems could be solved by creating an EVA Wiki using MediaWiki, a free and open-source software developed by the Wikimedia Foundation. The EVA Wiki developed into an EVA-specific Wikipedia on an internal NASA server. While the technical implementation of the wiki had many challenges, one of the biggest hurdles came from a cultural shift. Like many enterprise organizations, the EVA Operations group was accustomed to hierarchical data structures and individually-owned documents. Instead of sorting files into various folders, the wiki searches content. Rather than having a single document owner, the wiki harmonized the efforts of many contributors and established an automated revision controlled system. As the group adapted to the wiki, the usefulness of this single portal for information became apparent. It transformed into a useful data mining tool for EVA flight controllers and instructors, as well as hundreds of others that support the EVA. Program managers, engineers, astronauts, flight directors, and flight controllers in differing disciplines now have an easier-to-use, searchable system to find EVA data. This paper presents the benefits the EVA Wiki has brought to NASA's EVA community, as well as the cultural challenges it had to overcome.

Johnston, Stephanie S.↗
NASA NTRS2016

EVA Wiki - Transforming Knowledge Management for EVA Flight Controllers and Instructors

The EVA Wiki was recently implemented as the primary knowledge database to retain critical knowledge and skills in the EVA Operations group at NASA's Johnson Space Center by ensuring that information is recorded in a common, easy to search repository. Prior to the EVA Wiki, information required for EVA flight controllers and instructors was scattered across different sources, including multiple file share directories, SharePoint, individual computers, and paper archives. Many documents were outdated, and data was often difficult to find and distribute. In 2011, a team recognized that these knowledge management problems could be solved by creating an EVA Wiki using MediaWiki, a free and open-source software developed by the Wikimedia Foundation. The EVA Wiki developed into an EVA-specific Wikipedia on an internal NASA server. While the technical implementation of the wiki had many challenges, one of the biggest hurdles came from a cultural shift. Like many enterprise organizations, the EVA Operations group was accustomed to hierarchical data structures and individually-owned documents. Instead of sorting files into various folders, the wiki searches content. Rather than having a single document owner, the wiki harmonized the efforts of many contributors and established an automated revision controlled system. As the group adapted to the wiki, the usefulness of this single portal for information became apparent. It transformed into a useful data mining tool for EVA flight controllers and instructors, as well as hundreds of others that support EVA. Program managers, engineers, astronauts, flight directors, and flight controllers in differing disciplines now have an easier-to-use, searchable system to find EVA data. This paper presents the benefits the EVA Wiki has brought to NASA's EVA community, as well as the cultural challenges it had to overcome.

Johnston, Stephanie S.↗
NASA NTRS2016

EVA Wiki - Transforming Knowledge Management for EVA Flight Controllers and Instructors

The EVA (Extravehicular Activity) Wiki was recently implemented as the primary knowledge database to retain critical knowledge and skills in the EVA Operations group at NASA's Johnson Space Center by ensuring that information is recorded in a common, searchable repository. Prior to the EVA Wiki, information required for EVA flight controllers and instructors was scattered across different sources, including multiple file share directories, SharePoint, individual computers, and paper archives. Many documents were outdated, and data was often difficult to find and distribute. In 2011, a team recognized that these knowledge management problems could be solved by creating an EVA Wiki using MediaWiki, a free and open-source software developed by the Wikimedia Foundation. The EVA Wiki developed into an EVA-specific Wikipedia on an internal NASA server. While the technical implementation of the wiki had many challenges, the one of the biggest hurdles came from a cultural shift. Like many enterprise organizations, the EVA Operations group was accustomed to hierarchical data structures and individually-owned documents. Instead of sorting files into various folders, the wiki searches content. Rather than having a single document owner, the wiki harmonized the efforts of many contributors and established an automated revision control system. As the group adapted to the wiki, the usefulness of this single portal for information became apparent. It transformed into a useful data mining tool for EVA flight controllers and instructors, and also for hundreds of other NASA and contract employees. Program managers, engineers, astronauts, flight directors, and flight controllers in differing disciplines now have an easier-to-use, searchable system to find EVA data. This paper presents the benefits the EVA Wiki has brought to NASA's EVA community, as well as the cultural challenges it had to overcome.

Johnston, Stephanie↗
DOE OSTI2023

Accelerating Random Forest Classification on GPU and FPGA

Random Forests (RFs) are a commonly used machine learning method for classification and regression tasks spanning a variety of application domains, including bioinformatics, business analytics, and software optimization. While prior work has focused primarily on improving performance of the training of RFs, many applications, such as malware identification, cancer prediction, and banking fraud detection, require fast RF classification. In this work, we accelerate RF classification on GPU and FPGA. In order to provide efficient support for large datasets, we propose a hierarchical memory layout suitable to the GPU/FPGA memory hierarchy. We design three RF classification code variants based on that layout, and we investigate GPU- and FPGA-specific considerations for these kernels. Our experimental evaluation, performed on an Nvidia Xp GPU and on a Xilinx Alveo U250 FPGA accelerator card using publicly available datasets on the scale of millions of samples and tens of features, covers various aspects. First, we evaluate the performance benefits of our hierarchical data structure over the standard compressed sparse row (CSR) format. Second, we compare our GPU implementation with cuML, a machine learning library targeting Nvidia GPUs. Third, we explore the performance/accuracy tradeoff resulting from the use of different tree depths in the RF. Finally, we perform a comparative performance analysis of our GPU and FPGA implementations. Our evaluation shows that for high accuracy targets, our GPU implementation yields 5-9x speedup over CSR, and up to a 2x speedup over cuML.

FPGA, Xilinx FPGA, GPU, Random Forest classificati↗
DOE OSTI2022

LANL Contributions to pyiron: An integrated development environment for computational materials science

To support and accelerate the development of simulation protocols in atomistic modelling, we introduce an integrated development environment (IDE) for computational materials science called pyiron (http://pyiron.org). The pyiron IDE combines a web based source code editor, a job management system for build automation, and a hierarchical data management solution. The core components of the pyiron IDE are pyiron objects based on an abstract class, which links application structures such as atomistic structures, projects, jobs, simulation protocols and computing resources with persistent storage and an interactive user environment. The simulation protocols within the pyiron IDE are constructed using the Python programming language. To highlight key concepts of this tool as well as to demonstrate its ability to simplify the implementation and testing of simulation protocols we discuss two applications. In these examples we show how pyiron supports the whole life cycle of a typical simulation, seamlessly combines ab initio with empirical potential calculations, and how complex feedback loops can be implemented. While originally developed with focus on ab initio thermodynamics simulations, the concepts and implementation of pyiron are general thus allowing to employ it for a wide range of simulation topics.

Janssen, Jan↗
DOE OSTI2022

VERAIO Software Management Plan

VERAIO is a set of utility codes used to provide a common set of inputs and outputs to the Virtual Environment for Reactor Applications (VERA). VERA is a collection of several different computer codes that all have a common input and output. This prevents the need to manage input and output from each individual code, allowing for ease of use and reducing errors associated with code operability. The VERAIO utilities include VERAIn, VERAView, and VERARun. Each of these utilities is described below. VERAIn is an input processor that reads an ASCII input file generated by users, parses the file, performs some error checking, and writes an XML file to be read by other VERA codes. The main purpose of VERAIn is to provide a common input to all of the VERA codes, so users only need to learn one input. VERAIn is written in Perl and uses YAML configuration files to provide flexibility. VERAView is a graphical user interface (GUI) that reads a VERA hierarchical data format (HDF) output file and allows users to visualize results. VERAView is written in Python. VERARun is a script that drives the VERA execution in a high performance computing (HPC) environment. Work performed at the code level supports the quality assurance program plan (QAPP) (VERA-QA-001) and the VERA Software Quality Assurance Plan (VERA-QA-002).

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1986

Automatic discovery of optimal classes

A criterion, based on Bayes' theorem, is described that defines the optimal set of classes (a classification) for a given set of examples. This criterion is transformed into an equivalent minimum message length criterion with an intuitive information interpretation. This criterion does not require that the number of classes be specified in advance, this is determined by the data. The minimum message length criterion includes the message length required to describe the classes, so there is a built in bias against adding new classes unless they lead to a reduction in the message length required to describe the data. Unfortunately, the search space of possible classifications is too large to search exhaustively, so heuristic search methods, such as simulated annealing, are applied. Tutored learning and probabilistic prediction in particular cases are an important indirect result of optimal class discovery. Extensions to the basic class induction program include the ability to combine category and real value data, hierarchical classes, independent classifications and deciding for each class which attributes are relevant.

Cheeseman, Peter↗