Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data structures”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Optimizing Management of Persistent Data Structures in High-Performance Analytics

Large-scale data analytics workflows ingest massive input data into various data structures, including graphs and key-value datastores. These data structures undergo multiple transformations and computations and are typically reused in incremental and iterative analytics workflows. Persisting in-memory views of these data structures enables reusing them beyond the scope of a single program run while avoiding repetitive raw data ingestion overheads. Memory-mapped I/O enables persisting in-memory data structures without data serialization and deserialization overheads. However, memory-mapped I/O lacks the key feature of persisting consistent snapshots of these data structures for incremental ingestion and processing. The obstacles to efficient virtual memory snapshots using memory-mapped I/O include background writebacks outside the application’s control, and the significantly high storage footprint of such snapshots. To address these limitations, we present Privateer, a memory and storage management tool that enables storage-efficient virtual memory snapshotting while also optimizing snapshot I/O performance. Here, we integrated Privateer into Metall, a state-of-the-art persistent memory allocator for C++, and the Lightning Memory-Mapped Database (LMDB), a widely-used key-value datastore in data analytics and machine learning. Privateer optimized application performance by 1.22× when storing data structure snapshots to node-local storage, and up to 16.7× when storing snapshots to a parallel file system. Privateer also optimizes storage efficiency of incremental data structure snapshots by up to 11× using data deduplication and compression.

Computer science↗
DOE OSTI2026

Asi Nuclear Energy Sensors Data Portal Chatbot And Data Structuring Tool

The Idaho National Laboratory (INL) is advancing the development of an AI-powered chatbot and data structuring tool specifically designed to accelerate data mining processes for sensor-related information and seamlessly integrate the results into the ASI Sensors Data Portal (https://nes.energy.gov/). By doing so, the software aims to enhance the accessibility, usability, and organization of sensor data for nuclear energy applications. The software initial phase focuses on retrieving comprehensive datasets, prioritizing the past five years of publicly available information from the Office of Scientific and Technical Information (OSTI). These datasets will be meticulously processed to ensure compatibility, employing cleaning and preprocessing steps to eliminate irrelevant, incomplete, or corrupted information, thus establishing a robust foundation for subsequent AI use. The data will serve as the backbone for training an AI model and chatbot, which will act as an interactive tool enabling users to ask complex, context-specific questions and receive accurate, validated answers derived from constrained literature. In parallel, the project incorporates a data structuring process supported by AI to organize sensor information from multiple sources into a standardized format. This structured data will include detailed sensor specifications, such as measurement range, applications, accuracy, and operating conditions, generated and documented with AI. These specifications will be systematically integrated into the sensor portal. To maintain the highest levels of accuracy and relevance, all AI-generated outputs will be reviewed and validated by subject matter experts (SMEs), with additional fields or parameters added as needed. Future stages of the project aim to expand the dataset beyond OSTI to include other sources and potentially incorporate unclassified controlled information (UCI) with restricted access protocols to address security and confidentiality requirements.

Mapes, NormanJ. [Idaho National Laboratory (INL), ↗
DOE OSTI2025

Data Structure Alchemy

In an increasingly more data-driven world, the project set out to uncover the first principles of data-structure design, chart the immense design space they form, and build automation that can synthesize an optimal structure, or even a whole storage engine, for any given workload, hardware platform, and cost target. Data structures are at the center of every computational system and are directly responsible for its performance. Two core technical thrusts were defined: 1) Mapping design spaces for key data-centric abstractions (filters, hash functions, storage-engine layouts, neural-network topologies, blockchain protocols, image layouts, etc.). 2) Developing search & synthesis algorithms, initially analytical cost models, later neural-guided bi-level optimisers that navigate sextillions of candidate designs in seconds and materialise the best one as ready‐to-run code. This report distills the key insights, accomplishments, and impact.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Enriched immersed finite element and isogeometric analysis: algorithms and data structures

Immersed finite element methods provide a convenient analysis framework for problems involving geometrically complex domains, such as those found in topology optimization and microstructures for engineered materials. However, their implementation remains a major challenge due to, among other things, the need to apply nontrivial stabilization schemes and generate custom quadrature rules. This article introduces the robust and computationally efficient algorithms and data structures comprising an immersed finite element preprocessing framework. The input to the preprocessor consists of a background mesh and one or more geometries defined on its domain. The output is structured into groups of elements with custom quadrature rules formatted such that common finite element assembly routines may be used without or with only minimal modifications. The key to the preprocessing framework is the construction of material topology information, concurrently with the generation of a quadrature rule, which is then used to perform enrichment and generate stabilization rules. While the algorithmic framework applies to a wide range of immersed finite element methods using different types of meshes, integration, and stabilization schemes, the preprocessor is presented within the context of the extended isogeometric analysis. This method utilizes a structured B-spline mesh, a generalized Heaviside enrichment strategy considering the material layout within individual basis functions’ supports, and face-oriented ghost stabilization. Using a set of examples, the effectiveness of the enrichment and stabilization strategies is demonstrated alongside the preprocessor’s robustness in geometric edge cases. Additionally, the performance and parallel scalability of the implementation are evaluated.

Computer implementation↗
DOE OSTI2024

MemFriend: Understanding Memory Performance with Spatial-Temporal Affinity

In HPC applications, memory access behavior is one of the main factors affecting performance. Improving an application’s memory access behavior involves optimizing data layout and/or restructuring code, and requires studying spatial-temporal data locality. Existing data locality analyses focus on single-location metrics and are restricted to evaluating temporal locality. We introduce spatial-temporal affinity metrics that quantify temporal access proximity, forward access correlation, and nearby access correlation between pairs of memory locations. We describe methods for distinguishing between potential vs. realized affinity and for reasoning about affinity at multiple resolutions (3D, 2D, 1D). Finally, we construct spatial-temporal affinity signatures that classify memory behavior and that be used to reason about changes in software (data relayout, code refactoring) or hardware (caching, prefetching). We describe methods for signature visualization, interpretation, and quantitative comparison of signatures. We evaluate our methodology using applications with variants that contrast data structures, data layouts and algorithms. We show that spatial-temporal affinity analysis provides novel insights and enables predictive reasoning about application performance when contrasted with reuse distance analysis.

Suriyakumar, Yasodhadevi↗
DOE OSTI2025

Denoising Autoencoder for Reconstructing Sensor Observation Data and Predicting Evapotranspiration: Noisy and Missing Values Repair and Uncertainty Quantification

Abstract Machine learning (ML) methods applied in scientific research often deal with interrelated features in high‐dimensional data. Reducing data noise and redundancy is needed to increase prediction accuracy and efficiency especially when dealing with data from field sensors. We explored an unsupervised learning method, the denoising autoencoder (DAE), to extract the underlying data structure from noisy raw data in the context of predicting hydrologic quantities from multiple field sensors. These sensors have intrinsic instrumental noise and occasional malfunctions that cause missing values. Our DAE neural network reconstructed meteorological sensor data containing noise and missing values to predict evapotranspiration in a mountainous watershed. The DAE reconstructed the sensor variables with a mean coefficient of determination value of 0.77 across 15 dimensions representing individual sensors. It reduced variance and bias uncertainties compared to a classical autoencoder model. The reconstruction quality varied across dimensions depending on their cross‐correlation and alignment with the underlying data structure. Uncertainties arising from the model structure were overall higher than those resulting from data corruption. We attached the DAE structure to a downstream ET‐prediction neural network in three formats and achieved reasonably accurate ET predictions . The use of the DAE notably reduced variance uncertainty in ET prediction. However, excessive variance reduction may be accompanied by an increase in bias due to the intrinsic bias‐variance tradeoff. Our method of evaluating and reducing uncertainties in aggregated data from different sources can be used to improve predictive models, process understanding, and uncertainty quantification for better water resource management. Plain Language Summary We present a machine learning method, namely the denoising autoencoder, which reduces the effects of data noise and missing values typically present in scientific data sets collected through sensor measurements. This method selects the most relevant information from noisy raw data collected by the instruments and fills in missing values. To demonstrate the effectiveness of our method, we applied it to predict evapotranspiration, a hydrologic variable that represents the water moved from the land surface to the atmosphere through a combination of evaporation and plant water use (transpiration). We also used a random sampling technique (the Monte Carlo method) to compare the uncertainty in the predictions when using the raw and noisy data versus the reconstructed data. The denoising process produced more accurate predictions of evapotranspiration with less uncertainty. Improved predictions of evapotranspiration can lead to a better understanding and accounting of water budgets. This ML approach is broadly suitable for a wide variety of applications that involve noisy sensor data with missing values. Key Points We used a denoising autoencoder (DAE) neural network to reduce noise in meteorological and soil sensor observations by on average We used Monte Carlo sampling to estimate the bias and variance of all model outputs, including uncertainty sources from data and the model We attached the DAE component to a downstream neural network to predict ET with the variance reduced by , compared to that without the DAE

denoising autoencoder↗
DOE OSTI2026

Core Model Proposal #410: Updates to Socioeconomic and Macroeconomic Data, Processing Structure, and Visualization

This Core Model Proposal (CMP) comprehensively restructures and updates the macroeconomic and socioeconomic modules in gcamdata. It includes visualizations of key data inputs, accounting identities, and data flows in the context of GCAM-Macro-KLEM. Major improvements include: (1) updating the Penn World Table (PWT) to version 10 and incorporating a new source, the Global Macro Database (GMD); (2) updating the SSP socioeconomics database from version 3.0.1 to 3.2; (3) introducing SSP-specific differentiation of employment and labor force data; (4) improving data integration between national accounts (from PWT, GMD, and GTAP) and GDP/population data from external sources; and (5) general data cleaning and structural refinements. We document the data sources and key assumptions used throughout the processing. These updates establish the foundation for the forthcoming KLEAM version of GCAM-macro.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Informed unsupervised machine learning analysis of dislocation microstructure from high-resolution differential aperture X-ray structural microscopy data

This study leverages high-resolution differential-aperture X-ray structural microscopy (DAXM) to probe the local dislocation structure in deformed 304L-stainless steel at small strain, by measuring the lattice rotation and deviatoric elastic strain with a sub-micron resolution. For a single grain in a polycrystalline specimen, the measured lattice rotation field over the measured volume exhibited a multimodal distribution while the deviatoric elastic strain showed a single-mode distribution. An unsupervised Cauchy mixture machine learning model was developed to resolve the multimodal distribution of the lattice rotation. By mapping the lattice rotation data associated with each Cauchy peak in the model back onto the measured volume, we identify contiguous regions of the crystal rotated near the average values corresponding to the peaks of the overall rotation distribution. These regions represent the grain subdivision in the microstructure. Finally, the dislocation density tensor was also computed and its norm was laid over the rotation field to detect the subgrain boundaries. This step provided a validation of the Cauchy mixture model for the analysis of the lattice rotation distribution. The current study highlights the integration of advanced X-ray microscopy techniques with data-driven analysis methods to uncover detailed microstructure scales in deformed crystals.

Machine learning; Lattice rotation; High-energy X-↗
DOE OSTI2024

ISO_Fortran_binding_m v0.1.0

The Fortran programming language standard defines a broad feature set supporting the interoperability of Fortran programs with program written according to the C programming language standard. Among Fortran's C-interoperability features is a a C header file "ISO_Fortran_binding.h" This header file defines the interface to various C data structures and functions that C programs may use to access Fortran data entities. The ISO_Fortran_binding_m software defines a native Fortran module that presents an interface to these same data structures and functions. ISO_Fortran_bind_m thus enables Fortran programs to access and manipulate Fortran entities in ways that precisely mirror what C programs can do using ISO_Fortran_binding.h. ISO_Fortran_binding_m facilitates writing portable standard-conforming Fortran programs that emulate non-interoperable features, e.g., dynamic polymorphism, in a standard-conforming interoperable way similar but broader than what is demonstrated in Berkeley Lab's Caffeine software [1]. ISO_Fortran_binding_m also enables a Fortran programmer to extend Fortran's capabilities to emulate certain C functionality such as memory address arithmetic or computing C's "sizeof" function. [1] https://github.com/BerkeleyLab/caffeine/blob/213e3df1c319f0663306354414f352acda42a24f/src/caffeine/collective_subroutines/co_reduce_s.f90#L88 [2] https://github.com/BerkeleyLab/ISO_Fortran_binding_m/blob/0c585362bb4f2c72cf9049c800a7115b529ec533/src/iso_fortran_binding_m.F90#L196

Rouson, Damian↗
DOE OSTI2025

Data and scripts from: “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”

This data package includes data and scripts from the manuscript “Denoising autoencoder for reconstructing sensor observation data and predicting evapotranspiration: noisy and missing values repair and uncertainty quantification”.The study addressed common challenges faced in environmental sensing and modeling, including uncertain input data, missing sensor observations, and high-dimensional datasets with interrelated but redundant variables. Point-scaled meteorological and soil sensor observations were perturbed with noises and missing values, and denoising autoencoder (DAE) neural networks were developed to reconstruct the perturbed data and further predict evapotranspiration. This study concluded that (1) the reconstruction quality of each variable depends on its cross-correlation and alignment to the underlying data structure, (2) uncertainties from the models were overall stronger than those from the data corruption, and (3) there was a tradeoff between reducing bias and reducing variance when evaluating the uncertainty of the machine learning models.This package includes:(1) Four ipython scripts (.ipynb): “DAE_train.ipynb” trains and evaluates DAE neural networks, “DAE_predict.ipynb” makes predictions from the trained DAE models, “ET_train.ipynb” trains and evaluates ET prediction neural networks, and “ET_predict.ipynb” makes predictions from trained ET models.(2) One python file (.py): “methods.py” includes all user-defined functions and python codes used in the ipython scripts.(3) A “sub_models” folder that includes five trained DAE neural networks (in pytorch format, .pt), which could be used to ingest input data before being fed to the downstream ET models in ‘ET_train.ipynb” or ‘ET_predict.ipynb’.(4) Two data files (.csv). Daily meteorological, vegetation, and soil data is in “df_data.csv”, where “df_meta.csv” contains the location and time information of “df_data.csv”. Each row (index) in “df_meta.csv” corresponds to each row in “df_data.csv”. These data files are formatted to follow the data structure requirements and be directly used in the ipython scripts, and they have been shuffled chronologically to train machine learning models. The meteorological and soil data was collected using point sensors between 2019-2023 at(4.a) Three shrub-dominated field sites in East River, Colorado (named “ph1”, “ph2” and “sg5” in “df_meta.csv”, where “ph1” and “ph2” were located at PumpHouse Hillslopes, and “sg5” was at Snodgrass Mountain meadow) and(4.b) One outdoor, mesoscale, and herbaceous-dominated experiment in Berkeley, California (named “tb” in “df_meta.csv”, short for Smartsoils Testbed at Lawrence Berkeley National Lab).- See "df_data_dd.csv" and "df_meta_dd.csv" for variable descriptions and the Methods section for additional data processing steps. See "flmd.csv" and "README.txt" for brief file descriptions.- All ipython scripts and python files are written in and require PYTHON language software.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Electric Vehicle Charging Analytics and Reporting Tool (EV-ChART): Data Format and Preparation Guidance (V.5.0)

The Joint Office of Energy and Transportation maintains the Electric Vehicle Charging Analytics and Reporting Tool (EV-ChART), which provides a centralized hub for submitting electric vehicle (EV) charging infrastructure data directed by the Federal Highway Administration (23 CFR 680.112(a)-(c)). EV-ChART provides a streamlined data submission process and an integrated set of analytic tools, connects to other data sources, and empowers data sharing and access across stakeholders, including the public. Any data shared publicly will be aggregated and anonymized to stay in accordance with 23 CFR 680. This EV-ChART Data Format and Preparation Guidance provides a comprehensive overview of the data reporting requirements as authorized under 23 CFR 680.112(a)-(c)). The guidance is intended to be used alongside the EV-ChART Data Input Template, which defines the tabular data structure that these data submissions must follow. Per 23 CFR 680.112(a)-(c), the annual and quarterly data submissions are required of all National Electric Vehicle Infrastructure (NEVI) Formula Program projects, as well as projects for the construction of publicly accessible EV chargers that are funded with funds made available under Title 23, United States Code, including any EV charging infrastructure project funded with federal funds that is treated as a project on a federal-aid highway. One-time data submissions are required of both the NEVI Formula Program projects and grants awarded under 23 U.S.C. 151(f) for projects that are for EV charging stations located along and designed to serve the users of designated Alternative Fuel Corridors (AFCs). Other information and data required in 23 CFR 680, such as 23 CFR 680.112(d), 23 CFR 680.116(c), and 23 CFR 680.106(a), are not discussed in this guidance.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2024

On a Simplified Approach to Achieve Parallel Performance and Portability Across CPU and GPU Architectures

This paper presents software advances to easily exploit computer architectures consisting of a multi-core CPU and CPU+GPU to accelerate diverse types of high-performance computing (HPC) applications using a single code implementation. The paper describes and demonstrates the performance of the open-source C++ matrix and array (MATAR) library that uniquely offers: (1) a straightforward syntax for programming productivity, (2) usable data structures for data-oriented programming (DOP) for performance, and (3) a simple interface to the open-source C++ Kokkos library for portability and memory management across CPUs and GPUs. The portability across architectures with a single code implementation is achieved by automatically switching between diverse fine-grained parallelism backends (e.g., CUDA, HIP, OpenMP, pthreads, etc.) at compile time. The MATAR library solves many longstanding challenges associated with easily writing software that can run in parallel on any computer architecture. This work benefits projects seeking to write new C++ codes while also addressing the challenges of quickly making existing Fortran codes performant and portable over modern computer architectures with minimal syntactical changes from Fortran to C++. We demonstrate the feasibility of readily writing new C++ codes and modernizing existing codes with MATAR to be performant, parallel, and portable across diverse computer architectures.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

RNTuple: A CMS Perspective

ROOT is planning to move from TTree to RNTuple as the data storage format for HL-LHC in order to, for example, speed up the IO, make the files smaller, and have a modern C++ API. Initially, RNTuple was not planned to support the same set of C++ data structures as TTree supports. CMS has explored the necessary transformations in its standard persistent data types to switch to RNTuple. Many challenges were encountered as alternative data structures were explored. This contribution will discuss the challenges uncovered and how collaboration with the ROOT team allowed them to be overcome. The solution to the challenges allows progressive changes to the CMS data types rather than requiring a sudden change to all data types to be stored in RNTuple. Once the solution was achieved, storage performance comparisons using the CMS data types were possible between RNTuple and TTree. This contribution will also present the results of those comparisons.

Smith, Nick↗
DOE OSTI2025

Neutron capture measurements at UMass lowell research reactor

A new facility was designed around the thermal column beam port of the 1 MW Research Reactor at the University of Massachusetts Lowell. Thermal neutrons are collimated to a 1-inch diameter beam and incident on samples to induce the radiative neutron capture. New measurements of capture gamma rays are planned for Mn, Cu, Ni, Cr, and Gd samples in the next few years. The experiments will be carried out in close collaboration with the ENSDF (Evaluated Nuclear Structure Data File) evaluation group at Brookhaven National Laboratory. The gamma rays are measured using an array of high-resolution HPGe detectors. Some of the HPGe detectors use active Compton shields that were designed from scintillation detectors and improve significantly the signal-to-background in the measured gamma-ray spectra with HPGe. The experimental spectra will be validated by the GEANT4 simulations of the array and theoretical models of the emission of gamma rays from the compound nucleus. In conclusion, the new experimental results on the capture gamma ray intensities will be incorporated in future ENSDF evaluations.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Retrieval Augmented Generation for Robust Cyber Defense

In cybersecurity, the ability to efficiently analyze and respond to vulnerabilities, weaknesses, attack patterns, and threat tactics is critical for effective defense strategies. With the increasing complexity and volume of cybersecurity data, traditional methods of querying and retrieving information are often inadequate. To address this challenge, we implemented Retrieval-Augmented Generation (RAG) systems—CyRAG and GraphCyRAG—that integrate large language models (LLMs) with both structured data from relational databases and knowledge graphs such as Neo4j. CyRAG is designed to handle structured data, focusing on CVE (Common Vulnerabilities and Exposures) and CWE (Common Weakness Enumeration) entities to generate accurate and context-rich responses. In contrast, GraphCyRAG leverages Neo4j knowledge graphs to retrieve interconnected information from CVE, CWE, CAPEC (Common Attack Pattern Enumeration and Classification), and ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) datasets. By utilizing Neo4j’s graph-based framework, GraphCyRAG enables deeper traversal of relationships between vulnerabilities and attack patterns, providing cybersecurity analysts with more comprehensive insights into potential attack vectors and mitigation strategies. Our preliminary results demonstrate that integrating knowledge graphs with RAG significantly enhances both the accuracy and depth of threat analysis, allowing for the retrieval of dynamic, real-time data and the generation of contextually aware responses. This approach helps analysts uncover hidden relationships between cyber entities, predict exploit paths, and prioritize mitigation efforts effectively. The integration of RAG with cybersecurity knowledge graphs represents a significant advancement in cybersecurity threat intelligence, enabling more informed decision-making and stronger defense strategies.

97 MATHEMATICS AND COMPUTING↗