SEARCH · Engineering Papers

Results for “Benchmark data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

All sources NASA NTRS DOE OSTI

DOE OSTI2023

Benchmark Data Set of Crystalline Organic Semiconductors

This work reports a Benchmark Data set of Crystalline Organic Semiconductors to test calculations of the structural and electronic properties of these materials in the solid state. The data set contains 67 crystals consisting of mostly rigid molecules with a single dominant conformer, covering the majority of known structural types. The experimental crystal structure is available for the entire data set, whereas zero-temperature unit cell volume can be reliably estimated for a subset of 28 crystals. Using this subset, we benchmark r 2 SCAN-D3 and PBE-D3 density functionals. Then, for the entire data set, we benchmark approximate density functional theory (DFT) methods, including GFN1-xTB and DFTB3(3ob-3-1), with various dispersion corrections against r 2 SCAN-D3. Our results show that r 2 SCAN-D3 geometries are accurate within a few percent, which is comparable to the statistical uncertainty of experimental data at a fixed temperature, but the unit cell volume is systematically underestimated by 2% on average. The several times faster PBE-D3 provides an unbiased estimate of the volume for all systems except for molecules with highly polar bonds, for which the volume is substantially overestimated in correlation with the underestimation of atomic charges. Considered approximate DFT methods are orders of magnitude faster and provide qualitatively correct but overcompressed crystal structures unless the dispersion corrections are fitted by unit cell volume.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2025

MS25: Materials Science-Focused Benchmark Data Set for Machine Learning Interatomic Potentials

Here, we present MS25, a benchmark data set for evaluating machine learning interatomic potentials (MLIPs) across diverse materials-relevant systems including MgO surfaces, liquid water, zeolites, a catalytic Pt surface reaction, high-entropy alloys (HEAs), and disordered Zr-oxides. Five MLIP architectures (MACE, NequIP, Allegro, MTP, and Torch-ANI) are trained and tested, focusing not only on traditional metrics (energies, forces, and stresses) but also explicitly validating derived physical observables such as lattice constants, volumes, and reaction barriers. We find that most models reach comparable accuracy on standard error metrics across the simple systems, although equivariant MLIPs offer 1.5–2× improvements over nonequivariant MLIPs in energy and force error for structurally complex or compositionally disordered environments such as HEAs and Zr–O systems. Our analysis highlights that low errors in energy and force predictions do not guarantee reliable observables, emphasizing the necessity of explicit validation. We demonstrate limitations in cross-framework transferability, as models trained on one zeolite framework (CHA) fail to reliably generalize to predictions of structurally distinct frameworks (e.g., MFI). Size-extensive tests show some dependence on system size for MgO, resulting from forced periodicity. The HEA and Zr–O data sets are identified as challenging tests for future benchmarks and MLIP model architecture developments as they show significant differentiation in error between MLIP architectures and are still relatively difficult at 1000 training images. Moving forward, we recommend that benchmarking efforts shift their focus from marginal accuracy improvements in energy and force errors toward identifying and understanding model failure modes, rigorously assessing transferability, and evaluating how their errors affect observable predictions. For researchers looking to choose an MLIP architecture, we suggest selecting equivariant MLIP architectures if the complexity of the system is a challenge. For simple materials problems, auxiliary features such as integration with molecular dynamics engines, trade-offs between computational data set generation cost vs MLIP inference speed, and framework integration may play a more important decision factor than small differences in error metrics that are unlikely to matter for production-level research.

chemical structure↗

DOE OSTI2020

3D CFD Model Validation Using Benchmark Data of 1/16th Scaled VHTR Upper Plenum and Development of Wall Heat-Transfer Correlation For Laminar Flow

With support from the U.S. Department of Energy-Office of Nuclear Energy’s (DOE-NE’s) Nuclear Energy Advanced Modeling and Simulation (NEAMS) program, an effort has been pursued to support high-temperature gas-cooled reactor (HTGR) technology development and its modeling and simulation needs. There is a particular need for advanced modeling and simulation tools to predict thermal-fluid behavior in the nuclear reactor primary system, especially in the core and the lower and upper plena, during safety-related transients. In this report, two main such activities are presented relevant to the HTGRs: (1) three-dimensional (3D) computational fluid dynamics (CFD) validation using benchmark data from the upper plenum of Texas A&M University’s 1/16th scaled very-high-temperature gas-cooled reactor (VHTR), and (2) development of wall heat-transfer correlation for laminar flow in a wall-heated pipe. The CFD tool validation exercises can be helpful to choose the models and CFD tools to simulate and design specific components of the HTRGs such as upper plenum where jet mixing is a complex phenomenon. In a loss of forced circulation event, the laminar flow can be observed during the development of natural circulation flow. This work includes the development and validation of heat transfer correlations for laminar flow using the Nek5000 CFD code due to limited available experimental data for laminar flow conditions to guide low-order models (1D). In this report, the flow characteristics of a single isothermal jet discharging into the upper plenum was investigated using the Nek5000 Large-Eddy Simulation (LES) CFD tool. Several numerical simulations were performed for various jet-discharged Reynolds numbers ranging from 3,413 to 12,819. A grid-independent study was performed. The numerical results of mean velocity, root-mean-square fluctuating velocity, and Reynolds stress were compared against the benchmark data. Good agreement was obtained between simulated and measured data for axial mean velocities, except near the upper plenum hemisphere. The maximum predicted errors for axial mean velocities at various normalized coolant channel diameter heights of 1, 5, and 10 are 1.56%, 1.88%, and 3.82%, respectively. In addition, the predicted root-mean-square fluctuating velocity and Reynolds stress are qualitatively in agreement with the experimental data. The Nek5000 code was used to develop wall-heat transfer correlation for laminar flow in a cylindrical tube. Several simulations were performed for various Reynolds flow and wall-heat fluxes. A new heat transfer correlation was developed using data from Nek5000 simulation results and regression functions in Matlab. The developed heat transfer correlation is valid for various Reynolds flows from 200 to 2000. The predicted R² value for model fit was 0.875, which ensures that 87.5% of the model data lies on the Nek5000 data. Moreover, a machine learning (ML) tool was used to train and test the Nek5000 data. A good fit of the ML-based model was observed with the test data.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

DOE OSTI2023

An Inventory of AI-ready Benchmark Data for US Fires, Heatwaves, and Droughts

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2023

An Inventory of AI-ready Benchmark Data for US Fires, Heatwaves, and Droughts

Extreme weather events, including fires, heatwaves, and droughts, have significant impacts on earth, environmental, and energy systems. Mechanistic and predictive understanding, as well as probabilistic risk assessment of these extreme weather events, are crucial for detecting, planning for, and responding to these extremes. Records of extreme weather events provide an important data source for understanding present and future extremes, but the existing data needs preprocessing before it can be used for analysis. Moreover, there are many nonstandard metrics defining the levels of severity or impacts of extremes. In this study, we compile a comprehensive benchmark data inventory of extreme weather events, including fires, heatwaves, and droughts. The dataset covers the period from 2001 to 2020 with a daily temporal resolution and a spatial resolution of 0.5°×0.5° (~55km×55km) over the continental United States (CONUS), and a spatial resolution of 1km × 1km over the Pacific Northwest (PNW) region, together with the co-located and relevant meteorological variables. By exploring and summarizing the spatial and temporal patterns of these extremes in various forms of marginal, conditional, and joint probability distributions, we gain a better understanding of the characteristics of climate extremes. The resulting AI/ML-ready data products can be readily applied to ML-based research, fostering and encouraging AI/ML research in the field of extreme weather. This study can contribute significantly to the advancement of extreme weather research, aiding researchers, policymakers, and practitioners in developing improved preparedness and response strategies to protect communities and ecosystems from the adverse impacts of extreme weather events. Usage Notes We presented a long term (2001-2020) and comprehensive data inventory of historical extreme events with daily temporal resolution covering the separate spatial extents of CONUS (0.5°×0.5°) and PNW(1km×1km) for various applications and studies. The dataset with 0.5°×0.5° resolution for CONUS can be used to help build more accurate climate models for the entire CONUS, which can help in understanding long-term climate trends, including changes in the frequency and intensity of extreme events, predicting future extreme events as well as understanding the implications of extreme events on society and the environment. The data can also be applied for risk accessment of the extremes. For example, ML/AI models can be developed to predict wildfire risk or forecast HWs by analyzing historical weather data, and past fires or heateave , allowing for early warnings and risk mitigation strategies. Using this dataset, AI-driven risk assessment models can also be built to identify vulnerable energy and utilities infrastructure, imrpove grid resilience and suggest adaptations to withstand extreme weather events. The high-resolution 1km×1km dataset ove PNW are advantageous for real-time, localized and detailed applications. It can enhance the accuracy of early warning systems for extreme weather events, helping authorities and communities prepare for and respond to disasters more effectively. For example, ML models can be developed to provide localized HW predictions for specific neighborhoods or cities, enabling residents and local emergency services to take targeted actions; the assessment of drought severity in specific communities or watersheds within the PNW can help local authorities manage water resources more effectively.

Lin, Xinming↗

NASA NTRS2005

Benchmark Data for Evaluation of Aeroacoustic Propagation Codes With Grazing Flow

Increased understanding of the effects of acoustic treatment on the propagation of sound through commercial aircraft engine nacelles is a requirement for more efficient liner design. To this end, one of NASA s goals is to further the development of duct propagation and impedance reduction codes. A number of these codes have been developed over the last three decades. These codes are typically divided into two categories: (1) codes that use the measured complex acoustic pressure field to reduce the acoustic impedance of treatment that is positioned along the wall of the duct, and (2) codes that use the acoustic impedance of the treatment as input and compute the sound field throughout the duct. Clearly, the value of these codes is dependent upon the quality of the data used for their validation. Over the past two decades, data acquired in the NASA Langley Research Center Grazing Incidence Tube have been used by a number of researchers for comparison with their propagation codes. Many of these comparisons have been based upon Grazing Incidence Tube tests that were conducted to study specific liner technology components, and were incomplete for general propagation code validation. Thus, the objective of the current investigation is to provide a quality data set that can be used as a benchmark for evaluation of duct propagation and impedance reduction codes. In order to achieve this objective, two parallel efforts have been undertaken. The first of these is the development of an enhanced impedance eduction code that uses data acquired in the Grazing Incidence Tube. This enhancement is intended to place the benchmark data on as firm a foundation as possible. The second key effort is the acquisition of a comprehensive set of data selected to allow propagation code evaluations over a range of test conditions.

Jones, Michael G.↗

DOE OSTI2020

Alaskan carbon-climate feedbacks will be weaker than inferred from short-term manipulations: Alaskan Benchmark Data and Model runs

This submission aimed to assess differences in short-term step warming manipulations and long-term chronic response to climate change in Alaskan ecosystems. Briefly, climate warming is occurring fastest at high latitudes. Based on short-term field experiments, this warming is projected to stimulate soil organic matter decomposition, and promote a positive feedback to climate change. We show here that the tightly coupled, nonlinear nature of high-latitude ecosystems implies that short-term (< 10 year) warming experiments produce emergent ecosystem carbon stock temperature sensitivities inconsistent with emergent multi-decadal responses. We first demonstrate that a well-tested mechanistic ecosystem model accurately represents observed carbon cycle and active layer depth responses to short-term summer warming in four diverse Alaskan sites. We then show that short-term warming manipulations do not capture the non-linear, long-term dynamics of vegetation, and thereby soil organic matter, that occur in response to thermal, hydrological, and nutrient transformations belowground. Our results demonstrate significant spatial heterogeneity in multi-decadal Arctic carbon cycle trajectories and argue for more mechanistic models to improve predictive capabilities.The model used in the current study is available publicly (https://github.com/jinyun1tang/ECOSYS), and the current submission contains the python/ matlab codes for analyzing output from the model (includng a readme file to explain the codes). The benchmark data, also enclosed, was collected from a range of published and publicly available sources (extracted using GRABIT: https://www.mathworks.com/matlabcentral/fileexchange/7173-grabit). These sources describe warming induced changes in tundra/ boreal ecosystems.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS2003

Arithmetic Data Cube as a Data Intensive Benchmark

Data movement across computational grids and across memory hierarchy of individual grid machines is known to be a limiting factor for application involving large data sets. In this paper we introduce the Data Cube Operator on an Arithmetic Data Set which we call Arithmetic Data Cube (ADC). We propose to use the ADC to benchmark grid capabilities to handle large distributed data sets. The ADC stresses all levels of grid memory by producing 2d views of an Arithmetic Data Set of d-tuples described by a small number of parameters. We control data intensity of the ADC by controlling the sizes of the views through choice of the tuple parameters.

Frumkin, Michael A.↗

NASA NTRS1995

Effects of cavity dimensions, boundary layer, and temperature on cavity noise with emphasis on benchmark data to validate computational aeroacoustic codes

This report documents the results of an experimental investigation on the response of a cavity to external flowfields. The primary objective of this research was to acquire benchmark of data on the effects of cavity length, width, depth, upstream boundary layer, and flow temperature on cavity noise. These data were to be used for validation of computational aeroacoustic (CAA) codes on cavity noise. To achieve this objective, a systematic set of acoustic and flow measurements were made for subsonic turbulent flows approaching a cavity. These measurements were conducted in the research facilities of the Georgia Tech research institute. Two cavity models were designed, one for heated flow and another for unheated flow studies. Both models were designed such that the cavity length (L) could easily be varied while holding fixed the depth (D) and width (W) dimensions of the cavity. Depth and width blocks were manufactured so that these dimensions could be varied as well. A wall jet issuing from a rectangular nozzle was used to simulate flows over the cavity.

Ahuja, K. K.↗

DOE OSTI2025

DeepBench: A simulation package for physical benchmarking data

We introduce **DeepBench**, a python library that generates simple simulated image data from first principles, such as basic geometric shapes and astronomical objects. These data are highly valuable for developing (calibration, testing, and benchmarking) statistical and machine learning models because they make it possible to connect the final data product to physically interpretable inputs. This software includes tools to curate and store the datasets to maximize reproducibility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

NASA NTRSDate not supplied

Creating Benchmark Data for Artificial Intelligence and Machine Learning Space Biology Research

To identify an appropriate AI/ML approach for a specific problem, the best practice is to measure algorithm performance through the benchmarking process. A scientific benchmark consists of an AI-ready dataset and a reference implementation on a specific scientific question. The NASA Science Mission Directorate (SMD) has started the “Benchmark Initiative for AI/ML to create scientific benchmark datasets in three applications: 1) scientific benchmarking, which finds the best algorithm for a specific problem; 2) application benchmarking, which measures algorithm performance against a set of parameters; and 3) system benchmarking, which evaluates performance of hardware and software architecture. Currently, there are no standardized datasets available to benchmark AI/ML algorithms in the domain of space biology. In this work, we constructed two AI/ML-ready biological datasets from experiments in space-flown mice: cellular imaging and RNA-seq. First, radiation-exposed immune cells harbor DNA damage foci that can be fluorescently marked to visualize the amount of damage following exposure to ionizing radiation. However, such large datasets are difficult to analyze visually, due to imaging inconsistencies and human bias, and classical image processing approaches can fail on imaging artifacts. AI/ML are therefore exciting alternative, providing the speed of machines and the accuracy of humans. We have made this dataset available at https://registry.opendata.aws/bps_microscopy/. Second, high-throughput nucleic acid sequencing (DNA-seq, RNA-seq) has become widespread in biomedical research due to the growing availability and affordability of these assays. However, most sequencing datasets suffer from high dimensionality and low sample count. In this work, we used a generative adversarial network to synthesize a standardized, AI-ready, publicly available benchmark dataset for space biology RNA-seq data with sufficient space-flown and ground control mouse liver samples from NASA GeneLab. This dataset is available at https://registry.opendata.aws/bps_rnaseq/. These datasets are now fully open the Space Biology community to test their favorite AI/ML approaches.

James Casaletto↗

DOE OSTI2020

Identifying Questionable ICSBEP Benchmark Data and Underestimated Uncertainties using Machine Learning Methods [Slides]

Abstract not provided.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

DOE OSTI2022

The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider

We describe the outcome of a data challenge conducted as part of the Dark Machines (https://www.darkmachines.org) initiative and the Les Houches 2019 workshop on Physics at TeV colliders. The challenged aims to detect signals of new physics at the Large Hadron Collider (LHC) using unsupervised machine learning algorithms. First, we propose how an anomaly score could be implemented to define model-independent signal regions in LHC searches. We define and describe a large benchmark dataset, consisting of >1 billion simulated LHC events corresponding to 10\, fb^{-1} 10 f b − 1 of proton-proton collisions at a center-of-mass energy of 13 TeV. We then review a wide range of anomaly detection and density estimation algorithms, developed in the context of the data challenge, and we measure their performance in a set of realistic analysis environments. We draw a number of useful conclusions that will aid the development of unsupervised new physics searches during the third run of the LHC, and provide our benchmark dataset for future studies at https://www.phenoMLdata.org. Code to reproduce the analysis is provided at https://github.com/bostdiek/DarkMachines-UnsupervisedChallenge.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2024

A Simple Statistical Model for Randomized Benchmarking Data

Explore the source record for details and available documents.

Surti, Samyak Prasoon↗

NASA NTRS1984

Benchmark data on the separability among crops in the southern San Joaquin Valley of California

Landsat MSS data were input to a discriminant analysis of 21 crops on each of eight dates in 1979 using a total of 4,142 fields in southern Fresno County, California. The 21 crops, which together account for over 70 percent of the agricultural acreage in the southern San Joaquin Valley, were analyzed to quantify the spectral separability, defined as omission error, between all pairs of crops. On each date the fields were segregated into six groups based on the mean value of the MSS7/MSS5 ratio, which is correlated with green biomass. Discriminant analysis was run on each group on each date. The resulting contingency tables offer information that can be profitably used in conjunction with crop calendars to pick the best dates for a classification. The tables show expected percent correct classification and error rates for all the crops. The patterns in the contingency tables show that the percent correct classification for crops generally increases with the amount of greenness in the fields being classified. However, there are exceptions to this general rule, notably grain.

Morse, A.↗

NASA NTRS2015

Benchmark Data Set for Wheat Growth Models: Field Experiments and AgMIP Multi-Model Simulations.

The data set includes a current representative management treatment from detailed, quality-tested sentinel field experiments with wheat from four contrasting environments including Australia, The Netherlands, India and Argentina. Measurements include local daily climate data (solar radiation, maximum and minimum temperature, precipitation, surface wind, dew point temperature, relative humidity, and vapor pressure), soil characteristics, frequent growth, nitrogen in crop and soil, crop and soil water and yield components. Simulations include results from 27 wheat models and a sensitivity analysis with 26 models and 30 years (1981-2010) for each location, for elevated atmospheric CO2 and temperature changes, a heat stress sensitivity analysis at anthesis, and a sensitivity analysis with soil and crop management variations and a Global Climate Model end-century scenario.

climate change impact↗

NASA NTRSDate not supplied

Benchmark Data for Evaluation of NASA Impedance Eduction Methods

Results achieved with three acoustic liner configurations are used to evaluate the effects of recent modifications to the NASA Normal Incidence Tube and Grazing Flow Impedance Tube. These include a calibration liner, a wire mesh liner, and a perforate liner. The effects of source type, source level, and mean flow Mach number on the impedances educed with these liners are explored. Existing models are used to predict the impedance for each test condition and are compared with educed impedance spectra to determine their validity. The results suggest that educed impedances are similar for either a stepped sine or swept sine source. The predicted models are acceptable for the perforate liner and are quite good for the calibration and wire mesh liners.

acoustic↗

NASA NTRSDate not supplied

Benchmark Data for Evaluation of NASA Impedance Eduction Methods

Results achieved with three acoustic liner configurations are used to evaluate the effects of recent modifications to the NASA Normal Incidence Tube and Grazing Flow Impedance Tube. These include a calibration liner, a wire mesh liner, and a perforate liner. The effects of source type, source level, and mean flow Mach number on the impedances educed with these liners are explored. Existing models are used to predict the impedance for each test condition, and are compared with educed impedance spectra to determine their validity. The results suggest that educed impedances are similar for either a stepped sine or swept sine source. The predicted models are acceptable for the perforate liner, and are quite good for the calibration and wire mesh liners.

acoustic↗