Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “benchmark dataset”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

AI Benchmark Democratization and Carpentry

Benchmarks are a cornerstone of modern machine learning, enabling reproducibility, comparison, and scientific progress. However, AI benchmarks are increasingly complex, requiring dynamic, AI-focused workflows. Rapid evolution in model architectures, scale, datasets, and deployment contexts makes evaluation a moving target. Large language models often memorize static benchmarks, causing a gap between benchmark results and real-world performance. Beyond traditional static benchmarks, continuous adaptive benchmarking frameworks are needed to align scientific assessment with deployment risks. This calls for skills and education in AI Benchmark Carpentry. From our experience with MLCommons, educational initiatives, and programs like the DOE's Trillion Parameter Consortium, key barriers include high resource demands, limited access to specialized hardware, lack of benchmark design expertise, and uncertainty in relating results to application domains. Current benchmarks often emphasize peak performance on top-tier hardware, offering limited guidance for diverse, real-world scenarios. Benchmarking must become dynamic, incorporating evolving models, updated data, and heterogeneous platforms while maintaining transparency, reproducibility, and interpretability. Democratization requires both technical innovation and systematic education across levels, building sustained expertise in benchmark design and use. Benchmarks should support application-relevant comparisons, enabling informed, context-sensitive decisions. Dynamic, inclusive benchmarking will ensure evaluation keeps pace with AI evolution and supports responsible, reproducible, and accessible AI deployment. Community efforts can provide a foundation for AI Benchmark Carpentry.

von Laszewski, Gregor [Virginia U.]↗
DOE OSTI2021

Deep-Learning-Derived Evaluation Metrics Enable Effective Benchmarking of Computational Tools for Phosphopeptide Identification

Tandem mass spectrometry (MS/MS)-based phosphoproteomics is a powerful technology for global phosphorylation analysis. However, applying four computational pipelines to a typical mass spectrometry (MS)-based phosphoproteomic dataset from a human cancer study, we observed a large discrepancy among the reported phosphopeptide identification and phosphosite localization results, underscoring a critical need for benchmarking. While efforts have been made to compare performance of computational pipelines using data from synthetic phosphopeptides, evaluations involving real application data have been largely limited to comparing the numbers of phosphopeptide identifications due to the lack of appropriate evaluation metrics. We investigated three deep learning-derived features as potential evaluation metrics: phosphosite probability, Delta RT and spectral similarity. Predicted phosphosite probability is computed by MusiteDeep, which provides high accuracy as previously reported; Delta RT is defined as the absolute retention time (RT) difference between RTs observed and predicted by AutoRT; and spectral similarity is defined as the Pearson’s correlation coefficient between spectra observed and predicted by pDeep2. Using a synthetic peptide dataset, we found that both Delta RT and spectral similarity provided excellent discrimination between correct and incorrect peptide-spectrum matches (PSMs) both when incorrect PSMs involved wrong peptide sequences and even when incorrect PSMs were caused by only incorrect phosphosite localization. Based on these results, we used all the three deep learning-derived features as evaluation metrics to compare different computational pipelines on diverse set of phosphoproteomic datasets and showed their utility in benchmarking performance of the pipelines. The benchmark metrics demonstrated in this study will enable users to select computational pipelines and parameters for routine analysis of phosphoproteomics data and will offer guidance for developers to improve computational methods.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2026

Intern-Artificial Intelligence Benchmarking

Benchmarks provide a standardized method for evaluating different AI models, enabling reproducibility and comparison between models, and facilitating scientific progress. As AI models continue to develop rapidly, incorporating new datasets, capabilities, and architectures becomes more complicated. Therefore, the current static benchmarks become increasingly irrelevant. The MLCommons team argues that to make AI benchmarks more relevant, it involves making the benchmarks themselves more dynamic, as well as technical innovations that make it easier for scientists and researchers at all levels to use and contribute to the benchmarks. The current progress in technical innovation is a software that allows for a detailed view of a collection of AI benchmarks to be output in various formats that are easily readable and accessible.

Krishnan, Anjay [Fermilab]↗
DOE OSTI2026

Comparison of DeePMD, MTP, GAP, ACE and MACE Machine‐Learned Potentials for Radiation‐Damage Simulations: A User Perspective

Accurate and efficient interatomic potentials are essential for molecular dynamics (MD) simulations of radiation damage, gas diffusion, and phase stability in complex ceramics such as LiAlO 2 , especially under extreme conditions relevant to tritium production. Here, we evaluate the performance of six machine-learned interatomic potentials (MLIPs), moment tensor potential (MTP), Gaussian approximation potential, deep potential (DeePMD), atomic cluster expansion (ACE), message-passing ACE (multilayer atomic cluster expansion (MACE) pretrained) and MACE (trained from-scratch), all trained on the same density functional theory dataset with inclusion of tritium. The MLIPs are benchmarked against traditional Buckingham and ReaxFF potentials in terms of energy accuracy, density predictions, thermal equilibration behavior, threshold displacement energy (E d ), tritium diffusivity, and computational cost. Among the models, MTP shows the best overall balance between efficiency and accuracy, with low force and energy errors and realistic E d values for Li and Al. The ACE and MACE (pretrained and trained from scratch) models exhibit high E d (>200 eV) and unphysical pair interactions. DeePMD underestimates Ed due to overly repulsive behavior even at equilibrium distances. All models over-estimate tritium diffusion but the pretrained MACE model behaves well during tritium-diffusion simulations up to 500 K, maintaining diffusivities in the physically consistent 10 −11 m 2 /s range. Finally, we quantify the computational cost of each potential in large-scale atomic/molecular massively parallel simulator, finding that only MTP is more efficient than traditional empirical potentials, while others are significantly more expensive. These findings explain the trade-offs between accuracy and computational cost in MLIP development and provide essential guidance for use in high-throughput radiation damage and gas diffusion simulations in nuclear ceramics.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2022

When physics-informed data analytics outperforms black-box machine learning: A case study in thickness control for additive manufacturing

Aerosol jet printing (AJP) has emerged as a promising noncontact additive manufacturing method for high-resolution printing for a wide range of material systems. A key challenge limiting the broader adoption of AJP in the material science community is the lack of methods to precisely control thickness. Herein, we develop a model-based design of experiment (MBDoE) framework that integrates physics-informed models, nonlinear regression, and information criteria to postulate, select and calibrate the best model to describe and optimize the AJP manufacturing process. Starting with already available data from system commissioning (e.g., prior single variable sensitivity analysis), four candidate physics-informed models are postulated and trained. MBDoE identifies a single additional optimal experiment to validate these predictive models with quantified uncertainties, which are then used to determine the best experimental conditions to control printed film thickness. As a comparative benchmark, the analysis is repeated using the same dataset with nonparametric Gaussian process regression (GPR) model that does not incorporate physical information. Using MBDoE principles, we find that only five experiments are necessary to calibrate the nonlinear physics-informed parametric model, and with said limited data, this model outperforms the black-box machine learning GPR model. This key result underscores an emerging trend in the data science community: incorporating physical information into predictive models often drastically reduces the data requirements. Leveraging MBDoE further increased the data efficiency. By design, the proposed data science framework is general in nature and can be easily extended to other experimental and additive manufacturing systems beyond AJP.

Aerosol jet printing↗
DOE OSTI2021

Turbulence theories and statistical closure approaches

When discussing research in physics and in science more generally, it is common to ascribe equal importance to the three components of the scientific trinity: theoretical, experimental, and computational studies. This review will explore the future of modern turbulence theory by tracing its history, which began in earnest with Kolmogorov’s 1941 analysis of turbulence cascade and inertial range [A.N. Kolmogorov, Dokl. Akad. Nauk SSSR, 30, 299, (1941); 32, 19, (1941)]. The 80th Anniversary of Kolmogorov’s landmark study is a welcome opportunity to survey the achievements and evaluate the future of the theoretical approach of turbulence research. Over the years, turbulence theories have been critically important in laying the foundation of our understanding of the nature of turbulent flows. In particular, the Direct Interaction Approximation (DIA) [R.H. Kraichnan, J. Fluid Mech., 5, 497 (1959)] and its subsequent development, known as the statistical closure approach, can be identified as perhaps the most profound single advancement. The remarkable success of the statistical closure has furnished a platform to study such essential concepts as the energy transfer process and interacting scales, and the roles of the straining and sweeping motions. More recently, the quasi-Lagrangian formulation of V. L’vov & I. Procaccia and Kraichnan’s solvable passive scalar model provided powerful ways to explore another fundamental aspect of turbulent flows, the phenomena of intermittency, and the associated anomalous scaling exponents. In the meantime, the theory of fluid equilibria has been developed to describe the large-scale structures that can emerge from turbulent cascades of two-dimensional and geophysical flows at a later time. And yet, despite all these successes, analytical treatments suffer from mathematical complexities. As a result, the utility of theoretical approaches has been limited to relatively idealized flows. On the other hand, in recent decades, computational abilities and experimental facilities have reached an unprecedented scale. Looking beyond the horizon, the imminent deployment of exascale supercomputers will generate complete datasets of the entire flow field of key benchmark flows, allowing researchers to extract additional measurements concerning fully developed, complex turbulent flow fields far beyond those available from the statistical closure theories. Some other developments that could potentially influence the future course of turbulence theories include the advancement of machine learning, artificial intelligence, and data science; likely disruptions arising from the advent of quantum computation; and the increasingly prominent role of turbulence research in providing more accurate climate scientific data. Finally, turbulence theorists can leverage these developments by asking the right questions and developing advanced, sophisticated frameworks that will be able to predict and correlate vast amounts of data from the other two components of the trinity.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

Improving materials property predictions for graph neural networks with minimal feature engineering *

Graph neural networks (GNNs) have been employed in materials research to predict physical and functional properties, and have achieved superior performance in several application domains over prior machine learning approaches. Recent studies incorporate features of increasing complexity such as Gaussian radial functions, plane wave functions, and angular terms to augment the neural network models, with the expectation that these features are critical for achieving a high performance. Here, we propose a GNN that adopts edge convolution where hidden edge features evolve during training and extensive attention mechanisms, and operates on simple graphs with atoms as nodes and distances between them as edges. As a result, the same model can be used for very different tasks as no other domain-specific features are used. With a model that uses no feature engineering, we achieve performance comparable with state-of-the-art models with elaborate features for formation energy and band gap prediction with standard benchmarks; we achieve even better performance when the dataset size increases. Although some domain-specific datasets still require hand-crafted features to achieve state-of-the-art results, our selected architecture choices greatly reduce the need for elaborate feature engineering and still maintain predictive power in comparison.

42 ENGINEERING↗
DOE OSTI2021

Retrospective Analysis and Bayesian Model Averaging of CMIP6 Precipitation in the Nile River Basin

The Nile River basin is one of the global hotspots vulnerable to climate change impacts because of a fast-growing population and geopolitical tensions. Previous studies demonstrated that general circulation models (GCMs) frequently show disagreement in the sign of change in annual precipitation projections. Here, we first evaluate the performance of 20 GCMs from phase six of the Coupled Model Intercomparison Project (CMIP6) benchmarked against a high-spatial-resolution precipitation dataset dating back to 1983 from Precipitation Estimation from Remotely Sensed Information Using Artificial Neural Networks–Climate Data Record (PERSIANN-CDR). Next, a Bayesian model averaging (BMA) approach is adopted to derive probability distributions of precipitation projections in the Nile basin. Retrospective analysis reveals that most GCMs exhibit considerable (up to 64% of mean annual precipitation) and spatially heterogenous bias in simulating annual precipitation. Moreover, it is shown that all GCMs underestimate interannual variability; thus, the ensemble range is underdispersive and is a poor indicator of uncertainty. The projected changes from the BMA model show that the value and sign of change vary considerably across the Nile basin. Specifically, it is found that projected changes in the two headwaters basins, namely, the Blue Nile and Upper White Nile, are 0.03% and -1.65%, respectively; both are statistically insignificant at α = 0.05. The uncertainty range estimated from the BMA model shows that the probability of a precipitation decrease is much higher in the Upper White Nile basin whereas projected change in the Blue Nile is highly uncertain both in magnitude and sign of change.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

CISME report on experiments with HMX powder, PBX 9501 prills, and heated PBX 9501

The ability to predict post-ignition explosive response is relevant to explosives safety, particularly weapons assembly/disassembly operations that must operate under a sufficiently conservative safety basis to meet risk mitigation criteria for inadvertent nuclear detonation (IND). In particular, hazard determinations hinge on whether the PBX 9501 main charge can ultimately attain detonation from an initial, non-shock, ignition stimulus. The CISME experiment uses a spherical explosive charge, which is thermally ignited in the center using a fiber-delivered infrared laser pulse. This ignition mechanism - i.e. a localized thermal hotspot - is representative of the ignition-causing insults typical in accident scenarios, including low-velocity impact, drop, skid, and punch events. In fiscal year 2018 CISME experiments were conducted with pristine PBX 9501 spheres, confined and unconfined. Sub-detonative, low-violence reactions were observed. Here we report experiments conducted since 2018. The explosive charge was varied: experiments were conducted with room temperature pour-density HMX powder and prills of PBX 9501, and heated, thermally damaged PBX 9501. Experiments were conducted in a confined configuration, with two 4 mm-thick stainless-steel hemispherical shells bolted around the sphere. Detonation, via DDT (deflagration-to-detonation transition) was observed in the 3-inch diameter powdered HMX and the 6-inch diameter heated and thermally-damaged PBX 9501 tests. No transition to DDT was observed in smaller diameters for those configurations, and no DDT was observed for pour-density prills of PBX 9501 of any diameter. The original test series demonstrated that the response of pristine, undamaged PBX 9501 to a central ignition is relatively benign. This new test series shows that increased porosity, lack of polymer binders and thermal damage can greatly increase the violence of the post-ignition response, up to and including detonation. These results bridge the explosive response of PBX 9501 from sub-detonative through detonation as a function of diameter and provide valuable test cases for modeling efforts that strive to capture the DDT threshold. By exploring explosive response in a more representative 3D geometry, we can also benchmark these new results to more plentiful DDT datasets generated from testing in 1D tube confinement (the historical test configuration)–the configuration upon which almost all DDT models were developed.

42 ENGINEERING↗
DOE OSTI2026

AMVOS: Additive Manufacturing Video Object Segmentation Dataset

This dataset provides labeled video frames from four additive manufacturing (AM) processes for video object segmentation (VOS) tasks. It contains 90 video segments comprising 900 individually annotated frames across five AM datasets: laser hot-wire directed energy deposition (LHW-DED), tungsten inert gas wire arc additive manufacturing (TIG-WAAM), plasma arc welding (PAW), visible-light polymer extrusion (visPolymer), and near-infrared polymer extrusion (irPolymer). Each video segment consists of 10 contiguous frames with corresponding pixel-level object instance annotations. Depending on the process, two of four object classes are labeled per frame: Melt Pool, Feed Wire, Nozzle, or Material. Raw frames are provided as .jpg files and annotations as palettized .png files. The dataset follows the directory structure of established VOS benchmarks (DAVIS, YouTube-VOS, MOSE), enabling direct integration into VOS model training and evaluation pipelines for foundation model fine-tuning, domain adaptation, or zero-shot performance benchmarking. Data was collected at Oak Ridge National Laboratory's Manufacturing Demonstration Facility.

Wetzel, Jon [ORNL]↗
DOE OSTI2026

Intercomparison of flood inundation models across land use types and hydrological flood stages

Flood Inundation Mapping (FIM) model selection is a key operational decision because accurate, rapid mapping underpins early warning and resource allocation. FIM performance is context-dependent and can vary with hydrograph phase, land-use/land-cover (LULC), and the evaluation benchmark. Intercomparison studies typically assess a single near-peak snapshot against one reference dataset. Here, we provide a context-stratified intercomparison across (i) multiple hydrograph phases, (ii) LULC classes, and (iii) benchmark types, for five FIM approaches spanning a wide range of physical complexity and operational cost (TRITON, LISFLOOD-FP, HEC-RAS 2D, ARC-Curve2Flood, and OWP HAND-FIM). We use the Hurricane Matthew flood (2016) in the Neuse River Basin, North Carolina, USA, as a case study. Using high-resolution remote sensing-derived flood inundation maps, hand-labeled points, and building footprints, we assess model skill across two rising and two falling hydrograph limbs and across major LULC types. Results show that model rankings shift systematically across contexts: LISFLOOD-FP ranks highest in three of four flood phases, while TRITON leads during one rising limb phase; LISFLOOD-FP performs best in vegetated areas, whereas HEC-RAS improves relative performance in agricultural and urban areas; and benchmark choice influences conclusions, with LISFLOOD-FP performing best for flooded-building detection in the late falling limb, while TRITON ranks highest against hand-labeled points. We also report representative wall-clock runtimes for each workflow to provide use-case context for operational feasibility. Together, these results offer transferable guidance for model selection and for designing large-scale, benchmark-aware FIM intercomparison studies.

Nikrou, Parvaneh [University of Alabama]↗
DOE OSTI2021

Autotuning PolyBench benchmarks with LLVM Clang/Polly loop optimization pragmas using Bayesian optimization

Here, we develop a ytopt autotuning framework that leverages Bayesian optimization to explore the parameter space search and compare four different supervised learning methods within Bayesian optimization and evaluate their effectiveness. We select six of the most complex PolyBench benchmarks and apply the newly developed LLVM Clang/Polly loop optimization pragmas to the benchmarks to optimize them. We then use the autotuning framework to optimize the pragma parameters to improve their performance. The experimental results show that our autotuning approach outperforms the other compiling methods to provide the smallest execution time for the benchmarks syr2k, 3mm, heat-3d, lu, and covariance with two large datasets in 200 code evaluations for effectively searching the parameter spaces with up to 170,368 different configurations. We find that the Floyd-Warshall benchmark did not benefit from autotuning. To cope with this issue, we provide some compiler option solutions to improve the performance. Then we present loop autotuning without a user's knowledge using a simple mctree autotuning framework to further improve the performance of the Floyd-Warshall benchmark. We also extend the ytopt autotuning framework to tune a deep learning application.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

How efficiently can AI recognize Wireless Devices?

This poster presents a hardware benchmarking methodology for a 3-layer CNN waveform classifier deployed using ONNX Runtime on an NVIDIA Jetson AGX Orin. The dataset consist of 9 signal types, -30 to +30 dB SNR with 5dB increments. Benchmarking on the Jetson AGX Orin gave an accuracy of 91.9% and GPU throughput of 107,120 predictions/sec (23× faster than CPU). The Jetson GPU reached approximately 27M samples/sec with stable performance but fell below the 40 MHz rate needed for real-time radio feeds. Sustained testing of 5 minutes confirmed stable performance with no memory leaks, establishing a reproducible benchmarking baseline for future edge-deployment optimization.

99 - GENERAL AND MISCELLANEOUS↗
DOE OSTI2025

Fundamental limit of jet tagging

Identifying the origin of high-energy hadronic jets (jet tagging) has been a critical benchmark problem for machine learning in particle physics. Jets are ubiquitous at colliders and are complex objects that serve as prototypical examples of collections of particles to be categorized. Over the last decade, machine learning-based classifiers have replaced classical observables as the state of the art in jet tagging. Increasingly complex machine learning models are leading to increasingly more effective tagger performance. Our goal is to address the question of convergence—are we getting close to the fundamental limit on jet tagging or is there still potential for computational, statistical, and physical insights for further improvements? We address this question using state-of-the-art generative models to create a realistic, synthetic dataset with a known jet tagging optimum. Various state-of-the-art taggers are deployed on this dataset, showing that there is a significant gap between their performance and the optimum. Our dataset and software are made public to provide a benchmark task for future developments in jet tagging and other areas of particle physics.

Artificial intelligence↗
DOE OSTI2025

Cross-functional transferability in foundation machine learning interatomic potentials

The rapid development of foundation potentials (FPs) in machine learning interatomic potentials demonstrates the possibility for generalizable learning of the universal potential energy surface. The accuracy of FPs can be further improved by bridging the model from lower-fidelity datasets to high-fidelity ones. In this work, we analyze the challenge of this transfer learning (TL) problem within the CHGNet framework. We show that significant energy scale shifts and poor correlations between GGA and r 2 SCAN hinder cross-functional transferability. By benchmarking different TL approaches on the MP-r 2 SCAN dataset, we demonstrate the importance of elemental energy referencing in the TL of FPs. By comparing the scaling law with and without the pre-training on a low-fidelity dataset, we show that significant data efficiency can still be achieved through TL, even with a target dataset of sub-million structures. We highlight the importance of proper TL and multi-fidelity learning in creating next-generation FPs on high-fidelity data.

Huang, Xu [University of California, Berkeley, CA ↗
DOE OSTI2020

Automated detection of part quality during two-photon lithography via deep learning

Two-photon lithography (TPL) is an additive manufacturing technique for fabricating three-dimensional objects with nanoscale features. A main challenge of TPL is the routine and labor-intensive task of finding suitable light dosage parameters, i.e. writing speed and laser intensity that induce photo-polymerization within a wide variety of candidate photo-curing polymers. Another challenge is the monitoring required during fabrication. In this work, we apply machine learning (ML) models to accelerate the process of identifying optimal light dosage parameters and automate the detection of part quality. We curate TPL videos of different parts fabricated under a range of light dosage parameters using different resins and train spatial-temporal ML models on this data. Our results show that ML models can detect TPL part quality with a 95.1% accuracy in milliseconds. We also evaluate classification failures and identify two operating modes: parameter optimization and part quality detection. Last but not least, we publicly release this labelled dataset so that it may serve as a useful benchmark to the community. Our approach to process optimization and part quality detection addresses important aspects of TPL industrialization, is applicable beyond TPL and should benefit other additive manufacturing techniques with similar barriers to operating at industrial scale.

36 MATERIALS SCIENCE↗
DOE OSTI2022

A three-year dataset supporting research on building energy management and occupancy analytics

Abstract This paper presents the curation of a monitored dataset from an office building constructed in 2015 in Berkeley, California. The dataset includes whole-building and end-use energy consumption, HVAC system operating conditions, indoor and outdoor environmental parameters, as well as occupant counts. The data were collected during a period of three years from more than 300 sensors and meters on two office floors (each 2,325 m 2 ) of the building. A three-step data curation strategy is applied to transform the raw data into research-grade data: (1) cleaning the raw data to detect and adjust the outlier values and fill the data gaps; (2) creating the metadata model of the building systems and data points using the Brick schema; and (3) representing the metadata of the dataset using a semantic JSON schema. This dataset can be used in various applications—building energy benchmarking, load shape analysis, energy prediction, occupancy prediction and analytics, and HVAC controls—to improve the understanding and efficiency of building operations for reducing energy use, energy costs, and carbon emissions.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2025

A global dataset of terrestrial biological nitrogen fixation

Biological nitrogen fixation (BNF) is the main natural source of new nitrogen inputs in terrestrial ecosystems, supporting terrestrial productivity, carbon uptake, and other Earth system processes. We assembled a comprehensive global dataset of field measurements of BNF in all major N-fixing niches across natural terrestrial biomes derived from the analysis of 376 BNF studies. The dataset comprises 32 variables, including site location, biome type, N-fixing niche, sampling year, quantification method, BNF rate (kg N ha −1 y −1 ), the percentage of nitrogen derived from the atmosphere (%N dfa ), N fixer or N-fixing substrate abundance, BNF rate per unit of N fixer abundance, and species identity. Overall, the dataset combines 1,207 BNF rates for trees, shrubs, herbs, soil, leaf litter, woody litter, dead wood, mosses, lichens, and biocrusts, 152 herb %N dfa values, 1,005 measurements of N fixer or N-fixing substrate abundance, and 762 BNF rates per unit of N fixer abundance for a total of 424 species across 66 countries. This dataset facilitates synthesis, meta-analysis, upscaling, and model benchmarking of BNF fluxes at multiple spatial scales.

Reis Ely, Carla R. [Oregon State Univ., Corvallis,↗