Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Boosting Noise2Inverse via enhanced model selection for denoising computed tomography data

Synchrotron-based x-ray tomographic imaging enables the examination of the internal structure of materials at high spatial and temporal resolution. Experimental constraints can impose dose and time limits on the measurements, introducing a higher level of noise and artifacts in the reconstructed images. Deep learning has emerged as a powerful tool to remove noise from reconstructed images. Recently, the Noise2Inverse method was designed specifically for denoising reconstructed images without requiring paired noisy and clean images. This method creates multiple statistically independent reconstructions used to pair the data in which training involves transforming one reconstruction into the other, and vice versa. Originally designed to be used after a fixed number of epochs, we see in practice that this approach may not produce the optimal model and may unnecessarily waste computational resources. Therefore, we propose an alternative method of identifying the best model during training that aligns with the Noise2Inverse method. During validation, we compare the model output of the multiple reconstructions among each other. We hypothesize that the best model is the one that produces images with the highest similarity, implying a convergence in the predicted material properties and absorption values. To compare model outputs, we consider the absolute error, square error, structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and cosine similarity. We evaluate our method on two simulated tomography datasets and two, real-world, low-contrast, high-energy x-ray tomography datasets. We show our approach is more effective at determining the best model, up to an increase of 12.50% and 12.53% in SSIM and PSNR, respectively, while only requiring a fifth of the training time compared to the original approach.

CT↗
DOE OSTI2020

Understanding important features of deep learning models for segmentation of high-resolution transmission electron microscopy images

Cutting edge deep learning techniques allow for image segmentation with great speed and accuracy. However, application to problems in materials science is often difficult since these complex models may have difficultly learning meaningful image features that would enable extension to new datasets. In situ electron microscopy provides a clear platform for utilizing automated image analysis. In this work, we consider the case of studying coarsening dynamics in supported nanoparticles, which is important for understanding, for example, the degradation of industrial catalysts. By systematically studying dataset preparation, neural network architecture, and accuracy evaluation, we describe important considerations in applying deep learning to physical applications, where generalizable and convincing models are required. With a focus on unique challenges that arise in high-resolution images, we propose methods for optimizing performance of image segmentation using convolutional neural networks, critically examining the application of complex deep learning models in favor of motivating intentional process design.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Predicting the von Neumann entanglement entropy using a graph neural network

Calculating the von Neumann entanglement entropy from experimental data is challenging due to its dependence on the complete wavefunction, forcing reliance on approximations such as classical mutual information (MI). We propose a machine learning approach using a graph neural network to predict the von Neumann entropy directly from experimentally accessible bitstrings. We test this approach on a Rydberg ladder system and achieve a mean absolute error of $3.6\,\times 10^{-3}$ when evaluating within the training range on a dataset with entropy values ranging from 0 to 1.9. The model achieves a mean absolute percentage error of 1.44% and outperforms MI-based bounds. When tested beyond the training range, the model maintains reasonable accuracy. Furthermore, we demonstrate that fine-tuning the model with small datasets significantly improves performance on data outside the original training range.

graph neural networks↗
DOE OSTI2023

Mastering HPC Runtime Prediction: From Observing Patterns to a Methodological Approach

The continual expansion of high-performance computing (HPC) brings with it an increasing need for efficiency. Heavy investment in energy, hardware, and software infrastructure to support peta- and exascale computing requires the optimization of existing systems and, wherever possible, the discernment and adoption of best-practices towards these goals. Such is the case for runtime prediction. When a job is submitted to an HPC system, an estimate of its runtime is provided by the user in the form of "requested wallclock". Error in this user-provided estimate can lead to jobs being prematurely killed by the scheduler, increased wait time on the queue, and decreased system utilization. More than fifteen years of research has been directed at mitigating these effects by using data-driven runtime predictions. Codified here is a set of commonalities and insights emerging from this body of work, which we present as recommendations and best practices. These practices are combined into a methodological approach described and evaluated on an 11-million-job dataset from the National Renewable Energy Laboratory's petascale HPC system, Eagle. This dataset and the accompanying codebase have been released to the public domain for the benefit of the wider HPC research community.

high performance computing↗
DOE OSTI2021

Learning Universal Authorship Representations

This code contains all the utilities required to reproduce the results of our EMNLP 2021 paper "Learning Universal Authorship Representations". It contains the utilities required for download the datasets, training our model, and performing all evaluations necessary for reproducing the results in the paper. Here's the abstract of our work: Determining whether two documents were composed by the same author, also known as authorship verification, has traditionally been tackled using statistical methods. Recently, authorship representations learned using neural networks have been found to outperform alternatives, particularly in large-scale settings involving hundreds of thousands of authors. But do such representations learned in a particular domain transfer to other domains? Or are these representations inherently entangled with domain-specific features? To study these questions, we conduct the first large-scale study of cross-domain transfer for authorship verification considering zero-shot transfers involving three disparate domains: Amazon reviews, fanfiction short stories, and Reddit comments. We find that although a surprising degree of transfer is possible between certain do- mains, it is not so successful between others. We examine properties of these domains that influence generalization and propose simple but effective methods to improve transfer.

Rivera Soto, RafaelA↗
DOE OSTI2022

A Retrospective Analysis of Distributed Solar Interconnection Timelines and Related State Mandates

Most distributed residential photovoltaic (PV) systems must secure an interconnection agreement prior to operation that ensures the local electrical system will operate safely within the broader electrical grid. PV installers generally submit an interconnection application to the local utility that provides all the necessary details the utility needs to evaluate the proposed system. To encourage timely execution of this process, many states’ utility commissions have established mandates that limit the maximum number of days allowed for utility review and approval of interconnection for certain small, often residential applications. In this paper, we derive the median and range of cycle times for the pre-installation approval phase of the interconnection process (i.e., from application submission to approval by the utility) across 24 U.S. states, using a data set of approximately 170,000 projects. We evaluate the percentage of projects that are approved within respective state-mandated timelines from 2017– 2019. We further evaluate how timelines have evolved since 2012 for a subset of five states: Arizona, California, Colorado, New Jersey, and New York. The analyses are divided into two size ranges: systems that are ≤10 kilowatts (kW), and systems that are 11–50 kW.

14 SOLAR ENERGY↗
DOE OSTI2023

Cross-Scale Land-Atmosphere Experiment (CSLAEX) (Final Report)

It is now well recognized that land-atmosphere interactions play an important role in both weather and climate. These interactions take place over a wide range of spatial and temporal scales. Most of the current formulations of the momentum, heat and moisture transport laws are informed by observations with a limited temporal or spatial scale, and hence may have limited applicability. The objective of this work is to improve the multi-scale representation of the near-surface heat transfer in the ground and in the atmosphere using a combination of modeling and field observations. Our knowledge and predictive capacity in land-atmosphere coupling is limited by our capacity to observe the full-scale spectrum at play. In order to bridge this fundamental gap in our observation capacity I will work to: create the first long-term Integrated Cross-Scale land-Atmosphere Experiment (CSLAEX) since I will work to: measure the components of the surface energy balance at a spatial scale relevant to remote sensing and meteorological applications (5km), along with observations of the nested subscale processes with Distributed Temperature Sensors (DTS). This proposal aims to fill the gaps in modeling and observing land-atmosphere interactions across scales via a novel Cross-Scale Land-Atmosphere Experiment (CSLAEX) to be implemented as part of the Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Southern Great Plains (SGP) site near Lamont, Oklahoma. The proposed research effort aims at exploring land-atmosphere processes across multiple spatial scales and to investigate the effect of landscape heterogeneity using Distributed Temperature Sensor measurements. The novel instrumentation of the Surface Energy Balance and simultaneous observations of the atmospheric state (boundary layer, cloud, convection) available at the DOE ARM SGP site will be used to create CSLAEX. This dataset will be used to frame and evaluate new transport laws and parameterizations for the boundary layer and the land surface, which can be implemented into the Weather and Research Forecasting (WRF) model and Community Atmospheric Model (CAM). The result of this study should impact multiple fields: meteorology, climate, and hydrology.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

DetSuM: Detector Surrogate Model for LArTPC

In large neutrino experiments such as the Deep Underground Neutrino Experiment (DUNE), estimating detector response uncertainties typically requires simulation samples that consume substantial computing resources and time. To mitigate this challenge, we present DetSuM, an uncertainty-aware surrogate model designed to capture the detector response variations with reduced computing load compared to full simulations. This poster describes the construction and evaluation of DetSuM using simulation and reconstruction datasets in a rare-event search at DUNE. We assess DetSuM's ability to predict key detector-response variations and their associated uncertainties, discuss current limitations, and outline improvements to extend its validity in systematics studies of DUNE physics.

Li, Aobo [UC, San Diego]↗
DOE OSTI2023

An ensemble of 48 physically perturbed model estimates of the 1/8° terrestrial water budget over the conterminous United States, 1980–2015

Terrestrial water budget (TWB) data over large domains are of high interest for various hydrological applications. Spatiotemporally continuous and physically consistent estimations of TWB rely on land surface models (LSMs). As an augmentation of the operational North American Land Data Assimilation System Phase 2 (NLDAS-2) four-LSM ensemble, this paper describes a dataset simulated from an ensemble of 48 physics configurations of the Noah LSM with multi-physics options (Noah-MP). The 48 Noah-MP physics configurations are selected to give a representative cross-section of commonly used LSMs for parameterizing runoff, atmospheric surface layer turbulence, soil moisture limitation on photosynthesis, and stomatal conductance. The dataset spans from 1980 to 2015 over the conterminous United States (CONUS) at a monthly temporal resolution and a 1/8° spatial resolution. The dataset variables include total evapotranspiration and its constituents (canopy evaporation, soil evaporation, and transpiration), runoff (the surface and subsurface components), as well as terrestrial water storage (snow water equivalent, four-layer soil water content from the surface down to 2 m, and the groundwater storage anomaly). The dataset is available at https://doi.org/10.5281/zenodo.7109816. Evaluations carried out in this study and previous investigations show that the ensemble per forms well in reproducing the observed terrestrial water storage, snow water equivalent, soil moisture, and runoff. Noah-MP complements the NLDAS models well, and adding Noah-MP consistently improves the NLDAS es timations of the above variables in most areas of CONUS. Besides, the perturbed-physics ensemble facilitates the identification of model deficiencies. The parameterizations of shallow snow, spatially varying groundwater dynamics, and near-surface atmospheric turbulence should be improved in future model versions.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2003

Tropospheric Ozone Over a Tropical Atlantic Station in the Northern Hemisphere: Paramaribo, Surinam (6 deg N, 55 deg W)

We present an analysis of 2.5 years of weekly ozone soundings conducted at a new monitoring station in Paramaribo, Surinam (6 deg N,55 deg W). This is currently one of only three ozone sounding stations in the northern hemisphere (NH) tropics, and the only one in the equatorial Atlantic region. Paramaribo is part of the Southern Hemisphere ADditional Ozone Sounding program (SHADOZ). Due to its position close to the equator, the Inter Tropical Convergence Zone (ITCZ) passes over Paramaribo twice per year, which results in a semi-annual seasonality of many parameters including relative humidity and ozone. The dataset from Paramaribo is used to: (1) evaluate ozone variability relative to precipitation, atmospheric circulation patterns and biomass burning; (2) contrast ozone at the NH equatorial Atlantic with that at nearby southern hemisphere (SH) stations Natal (6 deg S,35 deg W) and Ascension (8 deg S,14 deg W); (3) compare the seasonality of tropospheric ozone with a satellite-derived ozone product: Tropical Tropospheric Ozone Columns from the Modified Residual method (MR-TTOC). We find that Paramaribo is a distinctly Atlantic station. Despite its position north of the equator, it resembles nearby SH stations during most of the year. Transport patterns in the lower and middle troposphere during February and March differ from SH stations, which leads to a seasonality of ozone with two maxima. MR-TTOC over Paramaribo does not match the observed seasonality of ozone due to the use of a SH ozone sonde climatology in the MR method. The Paramaribo ozone record is used to suggest an improvement for northern hemisphere MR-TTOC retrievals. We conclude that station Paramaribo shows unique features in the region, and clearly adds new information to the existing SHADOZ record.

Peters, W.↗
NASA NTRS2003

Accuracy and Precision in the Southern Hemisphere Additional Ozonesondes (SHADOZ) Dataset 1998-2000 in Light of the JOSIE-2000 Results

A network of 12 southern hemisphere tropical and subtropical stations in the Southern Hemisphere ADditional OZonesondes (SHADOZ) project has provided over 2000 profiles of stratospheric and tropospheric ozone since 1998. Balloon-borne electrochemical concentration cell (ECC) ozonesondes are used with standard radiosondes for pressure, temperature and relative humidity measurements. The archived data are available at:http: //croc.gsfc.nasa.gov/shadoz. In Thompson et al., accuracies and imprecisions in the SHADOZ 1998- 2000 dataset were examined using ground-based instruments and the TOMS total ozone measurement (version 7) as references. Small variations in ozonesonde technique introduced possible biases from station-to-station. SHADOZ total ozone column amounts are now compared to version 8 TOMS; discrepancies between the two datasets are reduced 2\% on average. An evaluation of ozone variations among the stations is made using the results of a series of chamber simulations of ozone launches (JOSIE-2000, Juelich Ozonesonde Intercomparison Experiment) in which a standard reference ozone instrument was employed with the various sonde techniques used in SHADOZ. A number of variations in SHADOZ ozone data are explained when differences in solution strength, data processing and instrument type (manufacturer) are taken into account.

Witte, J. C.↗
NASA NTRSDate not supplied

AstroLoc: An Efficient and Robust Localizer for a Free-flying Robot

We present AstroLoc, an efficient and robust monocular visual-inertial graph-based localization system used by the Astrobee free-flying robots onboard the International Space Station (ISS). We provide a novel localization system that limits the traditionally higher computation times for graph-based localization systems and enables the resource constrained Astrobee robots to benefit from their increased accuracy. We also introduce methods for handling cheirality issues for visual odometry and localization factors that further increase localization robustness. We evaluate the performance of AstroLoc on a dataset of ISS activities and show that it greatly improves pose, velocity, and IMU bias estimation accuracy while efficiently running in a limited computation environment. The source code for AstroLoc is released to the public.

Localization↗
NASA NTRSDate not supplied

Exercising Prediction Validation Methods for Analysis of Quiet Supersonic Overflight

En route noise certification procedures for quiet supersonic aircraft are in development under the International Civil Aviation Organization for a proposed standard that could enable civil supersonic overland flight. While these quiet supersonic aircraft do not exist yet, methods are being developed and evaluated using existing N-wave empirical datasets and low-boom simulation datasets. One potential aspect of future certification methods is the ability to use a validated prediction model to predict the certification noise level under reference day weather conditions. Methods to validate predictions, such as the two-sample t-test, Fisher’s combined probability test, two-sample Kolmogorov-Smirnov test, and equivalence testing, have been exercised and modified for this application. While some methods compare mean values, others compare values across the distribution, and a combination of methods can be used. The methods were applied to a subset of an N-wave dataset to demonstrate the applicability and identify potential areas for improvement. Atmospheric turbulence variability was included as a factor in these analyses through KZKFourier propagation simulations with parameters matching actual flights and weather conditions. Results are presented for six noise metrics (PL, ASEL, BSEL, DSEL, ESEL, and ISBAP), and plans to expand the analyses are discussed.

sonic boom↗
DOE OSTI2023

Mastering HPC Runtime Prediction: From Observing Patterns to a Methodological Approach: Preprint

The continual expansion of high-performance computing (HPC) brings with it an increasing need for efficiency. Heavy investment in energy, hardware, and software infrastructure to support peta- and exascale computing requires the optimization of existing systems and, wherever possible, the discernment and adoption of best-practices towards these goals. Such is the case for runtime prediction. When a job is submitted to an HPC system, an estimate of its runtime is provided by the user in the form of "requested wallclock''. Error in this user-provided estimate can lead to jobs being prematurely killed by the scheduler, increased wait time on the queue, and decreased system utilization. More than fifteen years of research has been directed at mitigating these effects by using data-driven runtime predictions. Codified here is a set of commonalities and insights emerging from this body of work, which we present as recommendations and best practices. These practices are combined into a methodological approach described and evaluated on an 11-million-job dataset from the National Renewable Energy Laboratory's petascale HPC system, Eagle. This dataset and the accompanying codebase have been released to the public domain for the benefit of the wider HPC research community.

high performance computing↗
DOE OSTI2026

Enhancing Operational Safety via Agentic Dialogue Hazard Identification Analysis

Operational safety in high-stakes domains such as industrial process control, autonomous, and safety-critical systems demand reliable hazard identification. While large language models (LLMs) have shown promise in automating safety analysis tasks, single-turn, monolithic inference is brittle: it lacks the self-correction, deliberation, and contextual refinement that safety engineers apply iteratively. In this paper, we introduce HAZDIAL, a framework that investigates whether structured agentic dialogue (multi-agent, multi-turn interactions) improves the quality of NLP-based hazard identification over single-pass baselines. We systematically compare two dialogue modalities: adversarial debate and constructive discussion, and propose an genetic algorithm-based agentic interaction optimization. We evaluate all configurations against a curated golden dataset using standard classification metrics (accuracy, precision, recall, F1) and a novel dialogue metrics. This work advances the intersection of dialogue systems, multi-agent reasoning, and AI safety, providing empirical evidence for dialogue-driven hazard analysis.

Das, Sanjay [ORNL] (ORCID:0009000542591915)↗
DOE OSTI2026

Dataset describing two reference models for full-spectral lighting and daylight simulations together with implementations for two software systems

A dataset of two spectral lighting simulation reference models - one office and one factory hall - is presented. It aims to demonstrate and support full-spectral daylight and electric lighting simulations and facilitate evaluation of non-visual effects of light. The dataset includes Rhino CAD geometry, comprehensive spectral material and light source data and window system BSDF data. Example implementations in the two software tools, Radiance and OWL, enable reproducible workflows and support adoption in other software. The dataset is openly available on Zenodo. The office model reproduces Room 518 at the University of Innsbruck, including a west-facing façade and interior furnishings. The factory hall model follows the proposed geometry in the European standard 15193 for building energy performance. Interior reflectances in the office were measured in-situ using a handheld spectrometer. Exterior spectra and factory hall materials matching specified reflectances were obtained from an online spectral materials database. Glazing transmittance was derived from IGDB data using LBNL Optics/WINDOW. BSDFs for venetian blinds at various tilt angles, and for a diffusing pane adapted from the Complex Glazing Database, were generated in WINDOW. Luminaires in both models are specified with photometric files (Eulumdat/IES) and lamp spectra (Fluorescent 840, 4000 K LED). The provided example implementations (Radiance, OWL) include prepared input data and scripts to run first spectral simulations; example results are also included. The dataset is prepared to support reuse by researchers, designers and software developers for method validation, software engineering and comparison, and development of spectral metrics and controls.

Geisler-Moroder, David↗
DOE OSTI2023

Continental Scale Hydrostratigraphy: Basin‐Scale Testing of Alternative Data‐Driven Approaches

Abstract Integrated hydrological modeling is an effective method for understanding interactions between parts of the hydrologic cycle, quantifying water resources, and furthering knowledge of hydrologic processes. However, these models are dependent on robust and accurate datasets that physically represent spatial characteristics as model inputs. This study evaluates multiple data‐driven approaches for estimating hydraulic conductivity and subsurface properties at the continental‐scale, constructed from existing subsurface dataset components. Each subsurface configuration represents upper (unconfined) hydrogeology, lower (confined) hydrogeology, and the presence of a vertical flow barrier. Configurations are tested in two large‐scale U.S. watersheds using an integrated model. Model results are compared to observed streamflow and steady state water table depth (WTD). We provide model results for a range of configurations and show that both WTD and surface water partitioning are important indicators of performance. We also show that geology data source, total subsurface depth, anisotropy, and inclusion of a vertical flow barrier are the most important considerations for subsurface configurations. While a range of configurations proved viable, we provide a recommended Selected National Configuration 1 km resolution subsurface dataset for use in distributed large‐and continental‐scale hydrologic modeling.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Time-Based CAN IDS Paper Results Code

Modern vehicles are complex cyber-physical systems made of hundreds of electronic control units (ECUs) that communicate over controller area networks (CANs). This inherited complexity has expanded the CAN attack surface which is vulnerable to message injection attacks. These injections change the overall timing characteristics of messages on the bus, and thus, to detect these malicious messages, time-based intrusion detection systems (IDSs) have been proposed. However, time-based IDSs are usually trained and tested on low-fidelity datasets with unrealistic, labeled attacks. This makes difficult the task of evaluating, comparing, and validating IDSs. Here we detail and benchmark four time-based IDSs against the newly published ROAD dataset, the first open CAN IDS dataset with real (non-simulated) stealthy attacks with physically verified effects. We found that methods that perform hypothesis testing by explicitly estimating message timing distributions have lower performance than methods that seek anomalies in a distribution related statistic. In particular, these “distribution-agnostic” based methods outperform “distribution-based” methods by at least 55% in area under the precision-recall curve (AUC-PR). Our results expand the body of knowledge of CAN time-based IDSs by providing details of these methods and reporting their results when tested on datasets with real advanced attacks. Finally, we develop an after-market plug-in detector using lightweight hardware, which can be used to deploy the best performing IDS method on nearly any vehicle.

Moriano, Pablo [Oak Ridge National Lab. (ORNL), Oa↗