Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Open Source Synergy: Developing and Validating PMU Data Analysis Techniques Using Open Source Tools and Datasets

This paper presents an exploration into the development and validation of data analysis approaches for Phasor Measurement Units (PMUs) using open-source datasets and tools. Various methods for event detection, event classification, frequency response, and oscillation analysis were tested. We leverage the capabilities of Archive Walker (AW), the Frequency Response Analysis Tool (FRAT), and the Oscillation Baselining and Analysis Tool (OBAT), all open-source tools, for efficient processing and analysis of synchrophasor data. The open-source Transmission Signature Library (TSL) dataset was employed as a dataset for a comprehensive evaluation to assess the performance and reliability of the proposed methods.

PMU, event analysis, oscillation, Frequency Respon↗
DOE OSTI2022

An Accurate, Error-Tolerant, and Energy-Efficient Neural Network Inference Engine Based on SONOS Analog Memory

In this work, we demonstrate SONOS (silicon-oxide-nitrideoxide- silicon) analog memory arrays that are optimized for neural network inference. The devices are fabricated in a 40nm process and operated in the subthreshold regime for in-memory matrix multiplication. Subthreshold operation enables low conductances to be implemented with low error, which matches the typical weight distribution of neural networks, which is heavily skewed toward near-zero values. This leads to high accuracy in the presence of programming errors and process variations. We simulate the end-to-end neural network inference accuracy, accounting for the measured programming error, read noise, and retention loss in a fabricated SONOS array. Evaluated on the ImageNet dataset using ResNet50, the accuracy using a SONOS system is within 2.16% of floating-point accuracy without any retraining. The unique error properties and high On/Off ratio of the SONOS device allow scaling to large arrays without bit slicing, and enable an inference architecture that achieves 20 TOPS/W on ResNet50, a >10× gain in energy efficiency over state-of-the-art digital and analog inference accelerators.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

CausalFS

SAND2021-4723 O CausalFS is a Python package that evaluates causal relationships between dataset features and the metric to be predicted. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Nguyen, Bernard↗
DOE OSTI2026

HydraGNN_GFM_FineTuning4Materials v1.0

This repository enables fine-tuning of the HydraGNN Predictive GFM 2026 — an open-source ensemble of pre-trained graph foundation models for atomistic materials modeling, developed at Oak Ridge National Laboratory. The GFM 2026 is freely available and downloadable via Globus from the OLCF Data Constellation (DOI: 10.13139/OLCF/2562660). Starting from these pre-trained weights, this repository provides a complete transfer learning pipeline for adapting the GFM ensemble to domain-specific molecular and materials property prediction tasks. It includes: 1) Utilities for ensemble fine-tuning with task-specific output heads 2) Example pipelines for eight widely-used materials and molecular datasets 3) Tools for model adaptation and head configuration 4) Data preprocessing utilities for each supported dataset 5) Benchmarking and evaluation scripts

Ungerboeck, Linda↗
DOE OSTI2026

VECTOR Phase 1 Dataset: CAV Trajectory and Energy Consumption Records

This dataset contains benchmark experimental data from Phase 1 of the VECTOR project, focusing on the energy impact of CAV hardware components. The dataset includes vehicle trajectory data (speed and position) and corresponding energy consumption records collected from a CAV platform equipped with lidar, cameras, onboard computation units, and communication modules. The primary objective is to quantify the baseline energy consumption attributable to sensing and computing systems, independent of any advanced cooperative control strategies. During experiments, the leading vehicle followed a predetermined velocity profile, and the following CAV mirrored this trajectory using a basic car-following control to ensure consistent driving behavior. This setup enables a reliable benchmark for assessing the energy cost introduced by onboard CDA hardware (e.g., lidar and GPU-based processing). The dataset is essential for evaluating energy baselines and supports future comparative studies involving additional cooperative strategies. ![system img](system.png) ![vector img](vector.png)

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2026

Depth-resolved sagebrush root metabolomics, rhizosphere microbial communities, and geochemistry at the East River Watershed

This data set consists of results from soil nutrient profile, untargeted metabolomics, mass spec imaging, and amplicon sequencing. Data for soil nutrient profile includes common cations (Ca, Mg, Na, and K etc.) extracted from 3 digesting steps – ammonia acetate (for exchangeable cations), nitric acid (for acid dissolved fraction), and hydrofluoric acid/perchloric acid (HF/HClO4) for whole soil digestion. It also includes concentration of organic carbon, inorganic nitrogen (ammonia and nitrate) and phosphorus (Bray-1 P and nitric acid extract), and total nitrogen and phosphorus. Data for untargeted metabolomics includes metabolomic profile for root exudate/tissues and soil extracts from depths at surface soil to saprolite, that were measured using gas chromatography – mass spectrometry (GC-MS), and liquid chromatography – tandem mass spectrometry (LC-MS/MS). Data for mass spec imaging includes spatial distribution of metabolites that were detected and annotated with Fourier transformation ion cyclotron resonance mass spectrometer (FTICR-MS). Data for amplicon sequencing includes the base paired 16S and ITS ribosomal RNA sequences from Miseq Illumina sequencing. All samples were collected from 2 sampling campaign October 2022 and June 2023. Collectively, these datasets enable a mechanistic evaluation of how nutrient acquisition, especially nitrogen and phosphorus, differs between shallow roots operating in soil and deep roots functioning within the fractured bedrock zone. All files are provided as comma-separated values (CSV) fies (.csv) and (GZIP) file (.gz). The compressed .gz FASTQ files can be read directly in R using the dada2 package as part of the amplicon sequence analysis workflow. This work was supported by the Watershed Function Science Focus Area at Lawrence Berkeley National Laboratory funded by the US Department of Energy, Office of Science, Biological and Environmental Research under Contract No. DE-AC02-05CH11231. This research was performed on a project award 60563 (https://dx.doi.org/10.46936/expl.proj.2022.60563/60008727) from the Environmental Molecular Sciences Laboratory, a DOE Office of Science User Facility sponsored by the Biological and Environmental Research program under Contract No. DE-AC05-76RL01830.

EARTH SCIENCE > AGRICULTURE > SOILS > CARBON↗
DOE OSTI2020

Towards Query-Efficient Black-Box Adversary with Zeroth-Order Natural Gradient Descent

Despite the great achievements of the modern deep neural networks (DNNs), the vulnerability/robustness of state-of-the-art DNNs raises security concerns in many application domains requiring high reliability. Various adversarial attacks are proposed to sabotage the learning performance of DNN models. Among those, the black-box adversarial attack methods have received special attentions owing to their practicality and simplicity. Black-box attacks usually prefer less queries in order to maintain stealthy and low costs. However, most of the current black-box attack methods adopt the first-order gradient descent method, which may come with certain deficiencies such as relatively slow convergence and high sensitivity to hyper-parameter settings. In this paper, we propose a zeroth-order natural gradient descent (ZO-NGD) method to design the adversarial attacks, which incorporates the zeroth-order gradient estimation technique catering to the black-box attack scenario and the second-order natural gradient descent to achieve higher query efficiency. The empirical evaluations on image classification datasets demonstrate that ZO-NGD can obtain significantly lower model query complexities compared with state-of-the-art attack methods.

Zhao, Pu↗
DOE OSTI2023

JetNet: A Python package for accessing open datasets and benchmarking machine learning methods in high energy physics

JetNet is a Python package that aims to increase accessibility and reproducibility for machinelearning (ML) research in high energy physics (HEP), primarily related to particle jets. Basedon the popular PyTorch ML framework, it provides easy-to-access and standardized interfacesfor multiple heterogeneous HEP datasets and implementations of evaluation metrics, lossfunctions, and more general utilities relevant to HEP.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

A library of AI-assisted FAIR water cycle and related disturbance datasets to enable model training, parameterization and validation

This whitepaper is responsive to focal area Data acquisition and assimilation enabled by machine learning, AI, and advanced methods. Here we describe how FAIR (Findable, Accessible, Reusable, Interoperable) datasets related to water cycle extremes are essential for successful implementation of ML in Earth System and other models. We also describe how AI can be used to acquire and integrate water cycle data related to extreme events to create a library of FAIR datasets for training and evaluating algorithms.

58 GEOSCIENCES↗
DOE OSTI2024

Seismic Signal Detection on International Monitoring System 3-Component Stations using PhaseNet

In this report we discuss training a deep learning seismic signal detection model on 3-component stations from the International Monitoring System (IMS) using the PhaseNet architecture. Using 14 years of associated signals from the International Data Centre’s (IDC) Late Event Bulletin (LEB), we auto-curated training data consisting of signal windows containing associated arrivals, and noise windows that contain no LEB-associated signals. We trained several models using different waveform window durations (30 seconds and 100 seconds), with and without bandpass filtering. We evaluated the effectiveness of our models using associated signals from the Unconstrained Global Event Bulletin (UGEB) and found that several of our models outperformed the signal detections from the IDC’s Selected Event List 3 (SEL3) arrival table. The SEL3 bulletin evaluated on the UGEB dataset with 100-second waveform windows registered a precision and recall of .15 and .48, respectively, versus .19 and .59 for our filtered-data model. For the 30-second waveform window dataset, the SEL3 bulletin achieved a precision and recall of .31 and .47, respectively, versus .32 and .60 for our filtered-data model. Finally, our models detected signals from all source-to-receiver distances, suggesting it is feasible to use a single PhaseNet model for the IMS network.

58 GEOSCIENCES↗
DOE OSTI2025

Machine learning guided selection of broad-spectrum epitope-specific functional antibodies for "Disease X"

Our project established and demonstrated a transfer learning framework that enables prediction of antibody–antigen interactions across related viruses. The approach focused on three major activities: 1. Conserved region and epitope identification – We compared viral protein structures and sequences to identify shared receptor-binding domains and neutralizing epitope regions across variants and related viruses. These conserved features formed the foundation for discovering broadly functional antibodies. 2. Machine learning model development – We built neural network–based models that integrate epitope features with antibody sequence information. Instead of relying solely on structural or physical properties, the models learned transferable patterns that describe antibody binding potential across different viral families. 3. Transfer learning and validation – Using SARS-CoV-2 and Ebola as source systems, we successfully transferred learned epitope features to predict antibody interactions for SARS CoV-1 and Marburg virus. Iterative cycles of dataset generation, retraining, and evaluation improved generalization and predictive power, ensuring the framework can adapt to new threats.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Comparison of metagenomes from fermentation of various agroindustrial residues suggests a common model of community organization

The liquid residue resulting from various agroindustrial processes is both rich in organic material and an attractive source to produce a variety of chemicals. Using microbial communities to produce chemicals from these liquid residues is an active area of research, but it is unclear how to deploy microbial communities to produce specific products from the different agroindustrial residues. To address this, we fed anaerobic bioreactors one of several agroindustrial residues (carbohydrate-rich lignocellulosic fermentation conversion residue, xylose, dairy manure hydrolysate, ultra-filtered milk permeate, and thin stillage from a starch bioethanol plant) and inoculated them with a microbial community from an acid-phase digester operated at the wastewater treatment plant in Madison, WI, United States. The bioreactors were monitored over a period of months and sampled to assess microbial community composition and extracellular fermentation products. We obtained metagenome assembled genomes (MAGs) from the microbial communities in each bioreactor and performed comparative genomic analyses to identify common microorganisms, as well as any community members that were unique to each reactor. Collectively, we obtained a dataset of 217 non-redundant MAGs from these bioreactors. This metagenome assembled genome dataset was used to evaluate whether a specific microbial ecology model in which medium chain fatty acids (MCFAs) are simultaneously produced from intermediate products (e.g., lactic acid) and carbohydrates could be applicable to all fermentation systems, regardless of the feedstock. MAGs were classified using a multiclass classification machine learning algorithm into three groups, organisms fermenting the carbohydrates to intermediate products, organisms utilizing the intermediate products to produce MCFAs, and organisms producing MCFAs directly from carbohydrates. This analysis revealed common biological functions among the microbial communities in different bioreactors, and although different microorganisms were enriched depending on the agroindustrial residue tested, the results supported the conclusion that the microbial ecology model tested was appropriate to explain the MCFA production potential from all agricultural residues.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2020

Characterization of organic aerosol across the global remote troposphere: a comparison of ATom measurements and global chemistry models

The spatial distribution and properties of submicron organic aerosol (OA) are among the key sources of uncertainty in our understanding of aerosol effects on climate. Uncertainties are particularly large over remote regions of the free troposphere and Southern Ocean, where very few data have been available and where OA predictions from AeroCom Phase II global models span 2 to 3 orders of magnitude, greatly exceeding the model spread over source regions. The (nearly) pole-to-pole vertical distribution of non-refractory aerosols was measured with an aerosol mass spectrometer onboard the NASA DC-8 aircraft as part of the Atmospheric Tomography (ATom) mission during the Northern Hemisphere summer (August 2016) and winter (February 2017). This study presents the first extensive characterization of OA mass concentrations and their level of oxidation in the remote atmosphere. OA and sulfate are the major contributors by mass to submicron aerosols in the remote troposphere, together with sea salt in the marine boundary layer. Sulfate was dominant in the lower stratosphere. OA concentrations have a strong seasonal and zonal variability, with the highest levels measured in the lower troposphere in the summer and over the regions influenced by biomass burning from Africa (up to 10 µg sm -3 ). Lower concentrations (~0.1–0.3 µg sm -3 ) are observed in the northern middle and high latitudes and very low concentrations (<0.1 µg sm -3 ) in the southern middle and high latitudes. The ATom dataset is used to evaluate predictions of eight current global chemistry models that implement a variety of commonly used representations of OA sources and chemistry, as well as of the AeroCom-II ensemble. The current model ensemble captures the average vertical and spatial distribution of measured OA concentrations, and the spread of the individual models remains within a factor of 5. These results are significantly improved over the AeroCom-II model ensemble, which shows large overestimations over these regions. However, some of the improved agreement with observations occurs for the wrong reasons, as models have the tendency to greatly overestimate the primary OA fraction and underestimate the secondary fraction. Measured OA in the remote free troposphere is highly oxygenated, with organic aerosol to organic carbon (OA/OC) ratios of ~2.2–2.8, and is 30 %–60 % more oxygenated than in current models, which can lead to significant errors in OA concentrations. The model–measurement comparisons presented here support the concept of a more dynamic OA system as proposed by Hodzic et al. (2016), with enhanced removal of primary OA and a stronger production of secondary OA in global models needed to provide better agreement with observations.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Climate Model Output Rewriter

The Climate Model Output Rewriter (CMOR) software was first developed by LLNL’s PCMDI program in early 2000s and was formally released with v1.0 (July 2006), v2.0 (January 2011), and v3.1(June 2016). CMOR is used to produce Climate and Forecast Convention (http://cfconventions.org/) CF-compliant netCDF files, in the standard format required to satisfy the World Climate Research Program (WCRP) Coupled Model Intercomparison Project (CMIP). The software has been used across multiple phases of the Earth System Modeling (ESM) project CMIP (CMIP3, CMIP5, CMIP6, and planned use in CMIP7) along with numerous parallel projects focused on preparation observations for use in model evaluation (obs4MIPs) and forcing datasets (input4MIPs) to guide ESM simulations to meet strict experimental protocols. More information can be obtained from the CMOR website and code repositories: https://cmor.llnl.gov/; https://github.com/pcmdi/cmor; https://github.com/PCMDI/cmor3_documentation The ESM variable definitions used as input for CMOR can also be viewed in code repositories: https://github.com/PCMDI/cmip3-cmor-tables/; https://github.com/PCMDI/cmip5-cmor-tables/; https://github.com/PCMDI/cmip6-cmor-tables/

Mauzey, ChristopherF↗
DOE OSTI2024

GOLEM: GOld standard for Learning and Evaluation of Motifs

Motifs are distinctive, recurring, widely used idiom-like words or phrases, often originating from folklore, whose meaning is anchored in a narrative and have a significance as communicative devices across a wide range of media, including news, literature, and propaganda. Many motifs concisely imply a large constellation of culturally relevant information, and their broad usage suggests their cognitive importance as touchstones of cultural knowledge. As such, their detection is a step towards culturally aware natural language processing. We present GOLEM (GOld standard for Learning and Evaluation of Motifs) a dataset of English news articles, opinion pieces, and broadcast transcripts annotated for motific information. The dataset identifies 25,737 motif candidates across 34 motif types drawn from three cultural or national groups: Jewish, Irish, and Puerto Rican. The dataset contains 2,024,141 words split into 25,737 text snippets drawn from 8,073 articles. Each motif candidate is labeled according to a scheme which identifies the type of usage (motific, referential, eponymic, or unrelated), resulting in 1,743 actual motific instances in the data. Annotation was performed by individuals identifying as members of each group and achieved a Fleiss’ kappa (?) of > 0.55. In addition to the data, we demonstrate that classification of the candidate type is a challenging task for Large Language Models (LLMs) using a few-shot approach; recent models such as T5, FLAN-T5, GPT-2, and Llama 2 (7B) achieved a performance of 41% accuracy at best, where the majority class accuracy is 41% and the average chance accuracy is 27%. These data will support development of new models and approaches for detecting (and reasoning about) motific information in text.

motif, culture, natural language, artificial intel↗
DOE OSTI2021

On the feasibility of using physics-informed machine learning for underground reservoir pressure management

In this work, we evaluate the feasibility of using physics-informed machine learning (PIML) for underground energy-related pressure management. To this end, we develop a PIML framework to manage underground reservoir pressures by training neural networks to determine fluid extraction rates for dedicated extraction wells during fluid injection operations given a range of reservoir conditions (e.g., transmissivity and storativity). We implement an automatically-differentiable analytical physics model of fluid flow in porous media within the PIML framework as a proxy for more complicated models. This allows us to execute a sufficient number of training scenarios to fully evaluate the feasibility of using PIML to support pressure management activities. We quantify the number of physics-model parameters required for automatic differentiation to become more efficient than finite-difference gradient calculations. We use a simple scenario with a single injector, extractor, and critical location for our feasibility analysis. We evaluate the effect of the size of the training dataset (i.e., the number of reservoir condition samples) on the accuracy and efficiency of the PIML framework. For an equivalent number of model evaluations, the larger training dataset took less time to train and produced a neural network that was able to more accurately manage reservoir pressures. We also evaluate the effect of the training dataset batch size (i.e., number of reservoir condition samples used to update the neural network coefficients during training; i.e., how the training dataset is partitioned). While training ran faster with larger batch sizes, they produced neural networks that managed pressures less accurately. We demonstrate the approach on a more complex scenario involving 10 injectors, 10 extractors, and 4 critical locations (a relatively high well density of 20 wells/km2). We provide the number of forward and adjoint model evaluations required in each case as an indication of the feasibility of using PIML for pressure management when more complicated physics models with longer execution times are used.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

A FAIR and AI-ready Higgs boson decay dataset

Abstract To enable the reusability of massive scientific datasets by humans and machines, researchers aim to adhere to the principles of findability, accessibility, interoperability, and reusability (FAIR) for data and artificial intelligence (AI) models. This article provides a domain-agnostic, step-by-step assessment guide to evaluate whether or not a given dataset meets these principles. We demonstrate how to use this guide to evaluate the FAIRness of an open simulated dataset produced by the CMS Collaboration at the CERN Large Hadron Collider. This dataset consists of Higgs boson decays and quark and gluon background, and is available through the CERN Open Data Portal. We use additional available tools to assess the FAIRness of this dataset, and incorporate feedback from members of the FAIR community to validate our results. This article is accompanied by a Jupyter notebook to visualize and explore this dataset. This study marks the first in a planned series of articles that will guide scientists in the creation of FAIR AI models and datasets in high energy particle physics.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

SRBench++: Principled Benchmarking of Symbolic Regression With Domain-Expert Interpretation

Symbolic regression searches for analytic expressions that accurately describe studied phenomena. The main promise of this approach is that it may return an interpretable model that can be insightful to users, while maintaining high accuracy. The current standard for benchmarking these algorithms is SRBench, which evaluates methods on hundreds of datasets that are a mix of real-world and simulated processes spanning multiple domains. At present, the ability of SRBench to evaluate interpretability is limited to measuring the size of expressions on real-world data, and the exactness of model forms on synthetic data. In practice, model size is only one of many factors used by subject experts to determine how interpretable a model truly is. Furthermore, SRBench does not characterize algorithm performance on specific, challenging sub-tasks of regression such as feature selection and evasion of local minima. In this work, we propose and evaluate an approach to benchmarking SR algorithms that addresses these limitations of SRBench by 1) incorporating expert evaluations of interpretability on a domain-specific task, and 2) evaluating algorithms over distinct properties of data science tasks. We evaluate 12 modern symbolic regression algorithms on these benchmarks and present an in-depth analysis of the results, discuss current challenges of symbolic regression algorithms and highlight possible improvements for the benchmark itself.

97 MATHEMATICS AND COMPUTING↗