Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “benchmark dataset”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32

scPlantAnnotate: an accurate and robust transformer-based model for plant cell type annotation

Accurate cell type annotation remains a major bottleneck in plant single-cell RNA sequencing (scRNA-seq), where existing tools are often adapted from animal studies and perform sub-optimally on plant data. The lack of plant-specific computational frameworks limits the construction of plant cell atlases and downstream biological discovery. We develop and evaluate scPlantAnnotate, a Transformer-based reference annotation framework tailored for plant scRNA-seq data, and benchmark it against state-of-the-art deep learning and conventional methods across multiple plant species. Species-specific scPlantAnnotate models were trained using curated datasets from Arabidopsis thaliana, Zea mays, Oryza sativa, and Glycine max. We compared scPlantAnnotate with leading baselines under both standard random-split evaluation and a more stringent leave-one-dataset-out setting, which tests robustness to completely unseen datasets and tissue types. scPlantAnnotate consistently outperforms existing approaches across all four species under random-split evaluation. In the leave-one-dataset-out setting for A. thaliana, where performance drops markedly for all methods due to strong batch effects and dataset heterogeneity, scPlantAnnotate nonetheless achieves the highest Accuracy, Macro-F1, Balanced Accuracy, and Macro-AUROC on average and ranks first on most held-out datasets. These results demonstrate improved robustness to dataset shifts, a critical yet underexplored challenge in plant scRNA-seq analysis. A freely accessible web server enables users to annotate their own datasets using pretrained models. scPlantAnnotate provides a plant-specific, Transformer-based framework for single-cell annotation that delivers state-of-the-art performance and enhanced robustness to unseen datasets. By addressing limitations of existing tools and enabling scalable reference-based annotation, scPlantAnnotate supports the development of comprehensive plant cell atlases and facilitates broader use of single-cell genomics in plant biology.

Bioinformatics↗

Neural Image Compression: Generalization, Robustness, and Spectral Biases

Recent advances in neural image compression (NIC) have resulted in models which are starting to outperform traditional codecs. While this has led to growing excitement about using these methods in real-world applications, the successful adoption of any machine learning system (including NIC) in the wild requires it to generalize (and be robust) to unseen distribution shifts at deployment time. Unfortunately, current research lacks comprehensive datasets and informative tools to evaluate and understand compression performance in real-world settings. To bridge this crucial gap, first, this paper presents a comprehensive benchmark suite to evaluate the out-of-distribution (OOD) performance of image compression methods. Specifically, we design CLIC-C and Kodak-C by introducing 15 common corruptions to popular CLIC and Kodak benchmarks. Next, we propose spectrally inspired introspection tools to gain a deeper understanding of errors introduced by image compression methods as well as their OOD performance. To this end, we carry out a detailed performance comparison of the classical codec with various variants of NIC (e.g., original, variable rate, pruned), revealing intriguing findings that challenge our current understanding of the strengths and limitations of NIC. Finally, we corroborate our empirical findings with theoretical analysis, providing an in-depth view of the OOD performance of NIC. Our benchmarks, spectral introspection tools, and findings provide a crucial bridge to the real-world adoption of NIC. We hope that our work will propel future efforts in designing more robust and generalizable NIC methods.

neural networks, Variational Autoencoder, robustne↗

Graph neural networks for CO 2 solubility predictions in Deep Eutectic Solvents

Deep Eutectic Solvents (DESs) are a promising class of solvents for CO 2 capture. DESs are complex mixtures that can be designed to optimize CO solubility and overall capture process efficiency. However, the vast design landscape of DES mixtures makes experimental investigation prohibitive; as such, there is a need for computational models that can quickly and efficiently navigate the design space and inform data collection efforts. In this work, we propose Graph Neural Network (GNN) models for predicting CO 2 solubility for DESs; the GNN leverages a mixture graph representation that captures the molecular structure of the DES components as well as their intermolecular interactions. Here, we compare the GNN framework against alternative architectures (neural networks, graph convolution networks, and random forests) and data representations (molecular fingerprints, sigma profiles, and graphs). We show that the proposed approach offers superior predictive performance; specifically, we show that solubility can be predicted reliably directly from molecular structure (without the need of using sigma profiles as proposed in previous studies). This result is important, as obtaining sigma profiles requires expensive density functional theory computations. We also explored the ability of GNNs to predict solubility for new DES mixtures and operating conditions. We found that the model extrapolates across temperature reliably. However, we also found deficiencies in the ability of the model to predict solubility for DES mixtures, pressures, and molar ratio not included in the training sets; we show that this is due to an inherent lack of chemical diversity in datasets available in the literature. The proposed computational capabilities can thus help navigate the design space of DES and inform data collection efforts. Our models, data, and benchmarks are shared as Python code implemented in Jupyter notebooks.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Vehicle Tracking with Crop-based Detection

End-to-end production of vehicle tracking data from video in real-time and with high accuracy remains a challenging problem due to the computational cost of object detection on each frame. In this work we present Tracking with Crop-based Detection, a method for speeding object tracking in constrained contexts (with stable cameras and relatively-predictable object motion) such as vehicle traffic monitoring. We leverage this context to provide a strong prior for object locations, which we use to 1.) boost detection speed by detecting objects only in regions corresponding to object priors on most frames and 2.) inform the selection of the detector output for each object. We evaluate Crop-based Detection as an extension to the KIOU object tracker (Crop-KIOU) on the UA-DETRAC dataset. The proposed tracker outperforms all other reported algorithms in terms of PR-MOTA, PR-MOTP, and mostly tracked objects on the UA-DETRAC benchmark, establishing a new state-of-the-art. Relative to tracking by detection with KIOU, Crop-KIOU achieves a 26% higher frame-rate and increases accuracy. Furthermore, Tracking with Crop-based Detection can be combined with frame skipping; we show a 149% increase in framerate relative to KIOU with no decrease in accuracy using this combination of methods.

Gloudemans, Derek↗

A Summary of Validation Studies for the Integrated TIGER Series Performed on ACORN Plus-up 218468/99

The Integrated TIGER Series (ITS) transport code is a valuable tool for photon-electron transport. A seven-problem validation suite exists to make sure that the ITS transport code works as intended. It is important to ensure that data from benchmark problems is correctly compared to simulated data. Additionally, the validation suite did not previously make use of a consistent quantitative metric for comparing experimental and simulated datasets. To this end, the goal of this long-term project was to expand the validation suite both in problem type and in the quality of the error assessment. To accomplish that, the seven validation problems in the suite were examined for potential drawbacks. When a drawback was identified, the problems were ranked based on severity of the drawback and approachability of a solution. We determined that meaningful improvements could be made to the validation suite by improving the analysis for the Lockwood Albedo problem and by introducing the Ross dataset as an eighth problem to the suite. The Lockwood error analysis has been completed and will be integrated in the future. The Ross data is unfinished, but significant progress has been made towards analysis.

97 MATHEMATICS AND COMPUTING↗

On Final Results From the Muon $g\mathrm{-}2$ Experiment at Fermilab

The Muon g-2 Experiment at Fermilab has measured the muon anomalous magnetic moment, a_mu, with unprecedented precision, leveraging a dataset from Runs 1 6 that is 21 times larger than its Brookhaven predecessor. This talk will present the experiment s final result, which serves as a benchmark for testing the Standard Model with high precision. The measurement was performed in a storage ring with a highly uniform magnetic field, where precise beam dynamics understanding is vital to determine the muon anomalous precession frequency. Key to this effort were advanced simulation tools including COSY INFINITY, precise fringe field modeling, and calculations of beam dynamics characteristics like tunes and chromaticity. We will conclude by exploring how the experiment's findings reshape our understanding of particle physics.

Valetov, Eremey [Michigan State U.]↗

Nuclear Dependence of Charged Current Inclusive $\bar{\nu}_\mu -$A Cross Section Measurements at 6 GeV at MINERvA

The main advantage of the MINERvA detector is that it uses differently sized nuclear targets, in the same neutrino beam, allowing for a direct study of nuclear medium effects, with reduced flux and detector uncertainties. This thesis presents the first high statistics charged current antineutrino-nucleus inclusive two-dimensional cross section measurements using an antineutrino beam with $\langle E_{\bar\nu}\rangle\sim$ 6 GeV, on iron, lead, carbon and hydrocarbon targets, as a function of bjorken $x$ and four-momentum transfer squared $Q^2$, invariant mass $W$ and $Q^2$ and muon longitudinal and transverse momenta, $p_t$ and $p_z$. These nuclei of different sizes were utilized to measure the nuclear dependence of the cross section. The antineutrino interactions that were utilized for the analysis presented in this thesis, were having an antimuon in the final state with energy 2-20GeV and with an antimuon scattering angle less than $17^o$. This analysis represents one of the largest antineutrino datasets analyzed to date in the energy region of $\langle E_{\bar\nu} \rangle \sim$ 6 GeV. This direct evidence for the nuclear medium effects, can help benchmarking models, which can be used by future neutrino experiments like DUNE, by extrapolating these results to Argon target.

Gaur, Prameet Kumar↗

An Open Combinatorial Diffraction Dataset Including Consensus Human and Machine Learning Labels with Quantified Uncertainty for Training New Machine Learning Models

Modern machine learning and autonomous experimentation schemes in materials science rely on accurate analysis of the data ingested by these models. Unfortunately, accurate analysis of the underlying data can be difficult, even for domain experts, complicating the training of the models intended to drive experiments. This is especially true when the goal is to identify the presence of weak signatures in diffraction or spectroscopic datasets. In this work, we examine a set of as-obtained diffraction data that track the phase transition from monoclinic to tetragonal in a Nb-doped VO2 film as a function of temperature and dopant concentration. We then task a set of domain experts and a set of machine learning experts with identifying which phase is present in each diffraction pattern manually and algorithmically, respectively; in both cases, the labels can vary dramatically, especially at the phase boundaries. We use the mode of the labels and the Shannon entropy as a method to capture, preserve and propagate consensus labels and their variance. Further we use the expert labels as a benchmark and demonstrate the use of Shannon entropy weighted scoring to test the performance of machine learning generated labels. Finally, we propose a material data challenge centered around generating improved labeling algorithms. This real-world dataset curated with expert labels can act as test bed for new algorithms. The raw data, annotations and code used in this study are all available online at data.gov and the interested reader is encouraged to replicate and improve the existing models

97 MATHEMATICS AND COMPUTING↗

Benchmarks for Model Validation based on LiDAR Wake Measurements

Technology development and design decisions in wind energy are often based on results from simulations performed for individual wind turbines or entire wind plants. It is therefore critical to ensure that the models being used for research and industry applications in wind energy be thoroughly validated against measurements. A full-system validation of wind plant simulations must consider the atmospheric inflow, the response of the wind turbines, and their wakes. This task is complicated by the lack of freely available, quality-controlled, high-quality measurements. Here, such measurements are used to offer a validation exercise that can be used to assess the accuracy of models of any fidelity level. When it comes to real-world measurements, the dataset considered herein is simple in terms of terrain but exhibits pronounced diurnal cycles. Instead of a full-scale wind plant, we consider an individual research-scale, utility wind turbine instrumented for power and loads measurements. Three benchmarks are defined, with increasing levels of complexity: near neutral, slightly unstable, and very stable atmospheric stratification. Through comparisons between observations and simulations, the benchmarks provide complementary information about the model performance and its ability to reproduce mean and dynamic wake characteristics. This article describes the measurements and methodology used to define these benchmarks and provides the information required to perform simulations and conduct the model-measurement comparison. The objective is to provide a robust wake model validation exercise open to anyone, which will serve to minimize uncertainty in model validation practices related to varying methodologies across simulation tools and users.

17 WIND ENERGY↗

Advances in building data management for building performance standards using the SEED platform

Reducing energy consumption and greenhouse gas emissions in the built environment is a critical step in achieving emission goals to mitigate climate change impacts. Local, federal, and international jurisdictions are deploying several methods to reduce energy and emissions such as voluntary and mandatory benchmarking and building performance standards, requiring building owners to reach energy and emission targets. Jurisdictions leveraging benchmarking and building performance standards require knowledge of the buildings covered; which is a large task due to staffing constraints, limited information on building characteristics and tax parcel data, and the need for advanced data management techniques to align datasets. This paper describes an open-source platform's recent advances to create consistent taxonomies, identify erroneous data, enable auditability, and track building performance. The paper concludes with two use cases on how the platform has been used by jurisdictions.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Systematic softening in universal machine learning interatomic potentials

Machine learning interatomic potentials (MLIPs) have introduced a new paradigm for atomic simulations. Recent advancements have led to universal MLIPs (uMLIPs) that are pre-trained on diverse datasets, providing opportunities for universal force fields and foundational machine learning models. However, their performance in extrapolating to out-of-distribution complex atomic environments remains unclear. In this study, we highlight a consistent potential energy surface (PES) softening effect in three uMLIPs: M3GNet, CHGNet, and MACE-MP-0, which is characterized by energy and force underprediction in atomic-modeling benchmarks including surfaces, defects, solid-solution energetics, ion migration barriers, phonon vibration modes, and general high-energy states. The PES softening behavior originates primarily from the systematically underpredicted PES curvature, which derives from the biased sampling of near-equilibrium atomic arrangements in uMLIP pre-training datasets. Our findings suggest that a considerable fraction of uMLIP errors are highly systematic, and can therefore be efficiently corrected. We argue for the importance of a comprehensive materials dataset with improved PES sampling for next-generation foundational MLIPs.

36 MATERIALS SCIENCE↗

Improving the Concrete Crack Detection Process via a Hybrid Visual Transformer Algorithm

Inspections of concrete bridges across the United States represent a significant commitment of resources, given their biannual mandate for many structures. With a notable number of aging bridges, there is an imperative need to enhance the efficiency of these inspections. This study harnessed the power of computer vision to streamline the inspection process. Our experiment examined the efficacy of a state-of-the-art Visual Transformer (ViT) model combined with distinct image enhancement detector algorithms. We benchmarked against a deep learning Convolutional Neural Network (CNN) model. These models were applied to over 20,000 high-quality images from the Concrete Images for Classification dataset. Traditional crack detection methods often fall short due to their heavy reliance on time and resources. This research pioneers bridge inspection by integrating ViT with diverse image enhancement detectors, significantly improving concrete crack detection accuracy. Notably, a custom-built CNN achieves over 99% accuracy with substantially lower training time than ViT, making it an efficient solution for enhancing safety and resource conservation in infrastructure management. These advancements enhance safety by enabling reliable detection and timely maintenance, but they also align with Industry 4.0 objectives, automating manual inspections, reducing costs, and advancing technological integration in public infrastructure management.

42 ENGINEERING↗

Machine learning and ligand binding predictions: A review of data, methods, and obstacles

We report that computational predictions of ligand binding is a difficult problem, with more accurate methods being extremely computationally expensive. The use of machine learning for drug binding predictions could possibly leverage the use of biomedical big data in exchange for time-intensive simulations. This paper reviews current trends in the use of machine learning for drug binding predictions, data sources to develop machine learning algorithms, and potential problems that may lead to overfitting and ungeneralizable models. A few popular datasets that can be used to develop virtual high-throughput screening models are characterized using spatial statistics to quantify potential biases. We can see from evaluating some common benchmarks that good performance correlates with models with high-predicted bias scores and models with low bias scores do not have much predictive power. A better understanding of the limits of available data sources and how to fix them will lead to more generalizable models that will lead to novel drug discovery.

59 BASIC BIOLOGICAL SCIENCES↗

Modeling Oxidative Dehydrogenation of Propane with Supported Vanadia Catalysts Using Multireference Methods

The oxidative dehydrogenation of propane over supported vanadium oxide catalysts poses significant computational challenges due to complex electronic structure changes along the reaction coordinate, driven primarily by changes in the oxidation states of vanadium. To address these challenges, we systematically test quantum chemical methods, including multireference (MR) approaches, domain-based local pair natural orbital coupled cluster theory (DLPNO-CCSD(T)), and density functional theory (DFT). The initial C–H bond-breaking transition state requires MR treatment due to its multireference character, while subsequent steps permit efficient single-reference calculations. For the rate-limiting C–H activation step mediated by the vanadyl moiety, complete 1 active space second-order perturbation theory (CASPT2) yields an apparent activation barrier (E app 600K ) of 138 kJ/mol, consistent with experimental values (134 ± 4 kJ/mol; Gruene et al. Catal. Today 2010, 157, 137). In contrast, DLPNO-CCSD(T) overestimates this barrier (198 kJ/mol), whereas DFT predictions span 125–150 kJ/mol, depending on the functional. Our multireference investigation of this transition metal oxide-catalyzed process demonstrates that an active space that incorporates the C–H σ and V=O σ/π bonding orbitals, oxygen lone pairs, and their antibonding counterparts adequately captures electronic structure changes along the chemical transformation. Furthermore, these findings provide a general strategy for active space selection in transition metal oxide-catalyzed C/O–H bond activation reactions. The reference dataset from this work, which includes MR calculations with manually selected active spaces for all intermediates and transition states in the propane ODH reaction network, will serve as a benchmark for automating active space selection in similar systems.

Catalysts↗

LandScan Mosaic Time Series Version 1.0: 1975-2024

The LandScan Mosaic Annual Global Ambient Population Time Series, Version 1.0, provides annual high-resolution global ambient population estimates from 1975 to 2024. The dataset is distributed as one Cloud-Optimized GeoTIFF per year on a globally aligned 3 arc-second WGS 84 grid. The 2024 layer is the LandScan Mosaic benchmark reference year, while the 1975–2023 layers are historical estimates generated using the LandCast backcasting framework.

Zimmer, Andrew [ORNL] (ORCID:0000000176838713)↗

Approaching coupled cluster accuracy with a general-purpose neural network potential through transfer learning

Computational modeling of chemical and biological systems at atomic resolution is a crucial tool in the chemist’s toolset. The use of computer simulations requires a balance between cost and accuracy: quantum-mechanical methods provide high accuracy but are computationally expensive and scale poorly to large systems, while classical force fields are cheap and scalable, but lack transferability to new systems. Machine learning can be used to achieve the best of both approaches. Here we train a general-purpose neural network potential (ANI-1ccx) that approaches CCSD(T)/CBS accuracy on benchmarks for reaction thermochemistry, isomerization, and drug-like molecular torsions. This is achieved by training a network to DFT data then using transfer learning techniques to retrain on a dataset of gold standard QM calculations (CCSD(T)/CBS) that optimally spans chemical space. The resulting potential is broadly applicable to materials science, biology, and chemistry, and billions of times faster than CCSD(T)/CBS calculations.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DeepBench: A simulation package for physical benchmarking data

We introduce **DeepBench**, a python library that generates simple simulated image data from first principles, such as basic geometric shapes and astronomical objects. These data are highly valuable for developing (calibration, testing, and benchmarking) statistical and machine learning models because they make it possible to connect the final data product to physically interpretable inputs. This software includes tools to curate and store the datasets to maximize reproducibility.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

A Subgrid Approach for Modeling Microtopography Effects on Overland Flow: Modeling Archive

This Modeling Archive is in support of an NGEE Arctic publication. A new subgrid model was implemented in the Advanced Terrestrial simulator (ATS) to capture micro-topography effects on surface flow. Model evaluation was performed on disparate landscapes including: 1) seven individual ice-wedge polygons and a cluster of ice-wedge polygons from Barrow Alaska, 2) hummocky microtopography from Spruce field site in Minnesota, 3) synthetic rills, and 4) synthetic multGaussian realizations. Results of the subgrid model are compared to benchmark (fine-scale) and null hypothesis (model ignoring microtopography). Our finding confirms that the effects of small-scale spatial heterogeneities can be captured in the coarsened models. The dataset contains variety of files including meshes, input files, and subgrid parameters required to run the simulations. Python scripts (jupyter notebooks) for post-processing and files for geometric analyses are also included. More details are provided in file "simulations_info", which has been provided in both .docx and .pdf formats. This dataset is a zip file.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗