Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 451 records · Page 25
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

The first ensemble of kilometer-scale simulations of a hydrological year over the third pole

An accurate understanding of the current and future water cycle over the Third Pole is of great societal importance, given the role this region plays as a water tower for densely populated areas downstream. An emerging and promising approach for skillful climate assessments over regions of complex terrain is kilometer-scale climate modeling. As a foundational step towards such simulations over the Third Pole, we present a multi-model and multi-physics ensemble of kilometer-scale regional simulations for the hydrological year of October 2019 to September 2020. The ensemble consists of 13 simulations performed by an international consortium of 10 research groups, configured with a horizontal grid spacing ranging from 2.2 to 4 km covering all of the Third Pole region. These simulations are driven by ERA5 and are part of a Coordinated Regional Climate Downscaling EXperiment Flagship Pilot Study on Convection-Permitting Third Pole. The simulations are compared against available gridded and in-situ observations and remote-sensing data, to assess the performance and spread of the model ensemble compared to the driving reanalysis during the cold and warm seasons. Although ensemble evaluation is hindered by large differences between the gridded precipitation datasets used as a reference over this region, we show that the ensemble improves on many warm-season precipitation metrics compared with ERA5, including most wet-day and hour statistics, and also adds value in the representation of wet spells in both seasons. As such, the ensemble will provide an invaluable resource for future improvements in the process understanding of the hydroclimate of this remote but important region.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Predictability of the early summer surface air temperature over Western South Asia

Variability of the Surface Air Temperature (SAT) over the Western South Asia (WSA) region leads to frequent heatwaves during the early summer (May-June) season. The present study uses the European Centre for Medium-Range Weather Forecast’s fifth-generation seasonal prediction system, SEAS5, from 1981 to 2022 based on April initial conditions (1-month lead) to assess the SAT predictability during early summer season. The goal is to evaluate the SEAS5’s ability to predict the El Niño-Southern Oscillation (ENSO) related interannual variability and predictability of the SAT over WSA, which is mediated through upper-level (200-hPa) geopotential height anomalies. This teleconnection leads to anomalously warm surface conditions over the region during the negative ENSO phase, as observed in the reanalysis and SEAS5. We evaluate SEAS5 prediction skill against two observations and three reanalyses datasets. The SEAS5 SAT prediction skill is higher with high spatial resolution observations and reanalysis datasets compared to the ones with low-resolution. Overall, SEAS5 shows reasonable skill in predicting SAT and its variability over the WSA region. Moreover, the predictability of SAT during La Niña is comparable to El Niño years over the WSA region.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Assessment of wind power scenario creation methods for stochastic power systems operations

Probabilistic scenarios of renewable energy production, such as wind, have been gaining popularity for use in stochastic variants of power systems operations scheduling problems, allowing for optimal decision-making under uncertainty. The quality of the scenarios has a direct impact on the value of the resulting decisions, but until now, methods for creating scenarios have not been compared under realistic operational conditions. Here, we compare the quality of scenario sets created using three different methods, based on a simulated re-enactment of stochastic day-ahead unit commitment and subsequent dispatch for a realistic test system. We create scenarios using a dataset of forecasted and actual wind power values, scaled to evaluate the effects of increasing wind penetration levels. We show that the choice of scenario set can significantly impact system operating cost, renewable energy use, and the ability of the system to meet demand. This result has implications for the ability of system operators to efficiently integrate renewable production into their day-ahead planning, highlighting the need for the use of performance-based assessments for scenario evaluation.

17 WIND ENERGY↗
DOE OSTI2025

Prediction of electric and magnetic fields from spectral data using machine learning algorithms for Doppler-free saturation spectroscopy diagnostics

The prediction of electric and magnetic field amplitudes from atomic spectral data is critical for plasma control in fusion devices such as tokamaks. Conventional approaches that rely on physics-based models are computationally expensive and unsuitable for real-time applications. In this work, we develop and benchmark three machine learning algorithms—simulation-based inference (SBI), fully connected neural networks (FCNN), and histogram-based gradient boosting regression (GBR-Hist)—to infer field intensities directly from Doppler-free saturation spectroscopy (DFSS) spectra. Synthetic datasets of spectra were generated using the EZSSS code and evaluated both with and without added Poisson noise to mimic experimental conditions. We find that SBI achieves the highest accuracy and robustness, FCNN provides a strong balance of accuracy and computational efficiency for real-time applications, and GBR-Hist offers the fastest inference but is more sensitive to noise. Furthermore, these results demonstrate the potential of machine learning to accelerate DFSS analysis and enhance its utility for plasma diagnostics and control.

Doppler-free saturation spectroscopy↗
DOE OSTI2021

Benchmarking graph neural networks for materials chemistry

Graph neural networks (GNNs) have received intense interest as a rapidly expanding class of machine learning models remarkably well-suited for materials applications. To date, a number of successful GNNs have been proposed and demonstrated for systems ranging from crystal stability to electronic property prediction and to surface chemistry and heterogeneous catalysis. However, a consistent benchmark of these models remains lacking, hindering the development and consistent evaluation of new models in the materials field. Here, we present a workflow and testing platform, MatDeepLearn, for quickly and reproducibly assessing and comparing GNNs and other machine learning models. We use this platform to optimize and evaluate a selection of top performing GNNs on several representative datasets in computational materials chemistry. From our investigations we note the importance of hyperparameter selection and find roughly similar performances for the top models once optimized. We identify several strengths in GNNs over conventional models in cases with compositionally diverse datasets and in its overall flexibility with respect to inputs, due to learned rather than defined representations. Meanwhile several weaknesses of GNNs are also observed including high data requirements, and suggestions for further improvement for applications in materials chemistry are discussed.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Soil carbon storage capacity of drylands under altered fire regimes

The determinants of fire-driven changes in soil organic carbon (SOC) across broad environmental gradients remains unclear, especially in global drylands. Here we combined datasets and field sampling of fire-manipulation experiments to evaluate where and why fire changes SOC and compared our statistical model to simulations from ecosystem models. Drier ecosystems experienced larger relative changes in SOC than humid ecosystems—in some cases exceeding losses from plant biomass pools—primarily explained by high fire-driven declines in tree biomass inputs in dry ecosystems. Many ecosystem models underestimated the SOC changes in drier ecosystems. Upscaling our statistical model predicted that soils in savannah–grassland regions may have gained 0.64 PgC due to net-declines in burned area over the past approximately two decades. Consequently, ongoing declines in fire frequencies have probably created an extensive carbon sink in the soils of global drylands that may have been underestimated by ecosystem models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Inference and reconstruction of the heimdallarchaeial ancestry of eukaryotes

Abstract In the ongoing debates about eukaryogenesis—the series of evolutionary events leading to the emergence of the eukaryotic cell from prokaryotic ancestors—members of the Asgard archaea play a key part as the closest archaeal relatives of eukaryotes 1 . However, the nature and phylogenetic identity of the last common ancestor of Asgard archaea and eukaryotes remain unresolved 2–4 . Here we analyse distinct phylogenetic marker datasets of an expanded genomic sampling of Asgard archaea and evaluate competing evolutionary scenarios using state-of-the-art phylogenomic approaches. We find that eukaryotes are placed, with high confidence, as a well-nested clade within Asgard archaea and as a sister lineage to Hodarchaeales, a newly proposed order within Heimdallarchaeia. Using sophisticated gene tree and species tree reconciliation approaches, we show that analogous to the evolution of eukaryotic genomes, genome evolution in Asgard archaea involved significantly more gene duplication and fewer gene loss events compared with other archaea. Finally, we infer that the last common ancestor of Asgard archaea was probably a thermophilic chemolithotroph and that the lineage from which eukaryotes evolved adapted to mesophilic conditions and acquired the genetic potential to support a heterotrophic lifestyle. Our work provides key insights into the prokaryote-to-eukaryote transition and a platform for better understanding the emergence of cellular complexity in eukaryotic cells.

Science & Technology - Other Topics↗
DOE OSTI2025

Low responsiveness of machine learning models to critical or deteriorating health conditions

Machine learning (ML) based mortality prediction models can be immensely useful in intensive care units. Such a model should generate warnings to alert physicians when a patient’s condition rapidly deteriorates, or their vitals are in highly abnormal ranges. Before clinical deployment, it is important to comprehensively assess a model’s ability to recognize critical patient conditions. We develop multiple medical ML testing approaches, including a gradient ascent method and neural activation map. We systematically assess these machine learning models’ ability to respond to serious medical conditions using additional test cases, some of which are time series. Guided by medical doctors, our evaluation involves multiple machine learning models, resampling techniques, and four datasets for two clinical prediction tasks. We identify serious deficiencies in the models’ responsiveness, with the models being unable to recognize severely impaired medical conditions or rapidly deteriorating health. For in-hospital mortality prediction, the models tested using our synthesized cases fail to recognize 66% of the injuries. In some instances, the models fail to generate adequate mortality risk scores for all test cases. Our study identifies similar kinds of deficiencies in the responsiveness of 5-year breast and lung cancer prediction models. Using generated test cases, we find that statistical machine-learning models trained solely from patient data are grossly insufficient and have many dangerous blind spots. Most of the ML models tested fail to respond adequately to critically ill patients. How to incorporate medical knowledge into clinical machine learning models is an important future research direction.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2021

Exploring Autoencoder-based Error-bounded Compression for Scientific Data

Error-bounded lossy compression is becoming an indispensable technique for the success of today's scientific projects with vast volumes of data produced during the simulations or instrument data acquisitions. Not only can it significantly reduce data size, but it also can control the compression errors based on user-specified error bounds. Autoencoder (AE) models have been widely used in image compression, but few AE-based compression approaches support error-bounding features, which are highly required by scientific applications. To address this issue, we explore using convolutional autoencoders to improve error-bounded lossy compression for scientific data, with the following three key contributions. (1) We provide an in-depth investigation of the characteristics of various autoencoder models and develop an error-bounded autoencoder-based framework in terms of the SZ model. (2) We optimize the compression quality for main stages in our designed AE-based error-bounded compression framework, fine-tuning the block sizes and latent sizes and also optimizing the compression efficiency of latent vectors. (3) We evaluate our proposed solution using five real-world scientific datasets and comparing them with six other related works. Experiments show that our solution exhibits a very competitive compression quality from among all the compressors in our tests. In absolute terms, it can obtain a much better compression quality (100%similar to 800% improvement in compression ratio with the same data distortion) compared with SZ2.1 and ZFP in cases with a high compression ratio.

Liu, Jinyang↗
DOE OSTI2022

A Framework for Neural Network Inference on FPGA-Centric SmartNICs

FPGA-based SmartNICs offer great potential to significantly improve the performance of high-performance computing and warehouse data processing by tightly coupling support for reconfigurable data-intensive computation with cross-node communication, thereby mitigating the von Neumann bottleneck. Existing work, however, has been generally been limited in that it assumes an accelerator model where kernels are offloaded to SmartNICs, but most control tasks are left to the CPUs. This leads to frequent waiting, inferior performance, and scaling challenges. In this work, we propose a new distributive data-centric computing framework, named FCsN, for reconfigurable SmartNIC-based systems. Through a lightweight task circulation execution model and its implementation architecture, FCsN allows the complete detaching of kernel execution, control logic, system scheduling, and network communication to the SmarNICs. This boosts performance by: (i) avoiding the control dependency with CPUs and (ii) supporting streaming kernel execution and network communication at line rate and in a very fine-grained manner. We demonstrate the efficiency and flexibility of FCsN using various types of neural network applications including graph neural networks; as these last are both irregular and data intensive they offer an especially robust demonstration. Evaluations using commonly-used neural network models and graph datasets show that a system with the support of FCsN can achieve, on average, 144 speedups over the MPI-based standard CPU baselines.

Guo, Anqi↗
DOE OSTI2024

A Deep Learning Approach for In-Network Synchrophasor Missing Data Recovery Using Programmable Network Switches

Phasor measurement unit (PMU) networks deliver accurate and timely measurements, which is essential for managing today’s electric power systems. To ensure data quality and enhance the cyber-resilience of PMU networks against malicious attacks and data errors, this study presents an online PMU missing data recovery scheme by leveraging P4 programmable switches. The data plane incorporates a customized PMU protocol parser that abstracts the necessary payload data for recovery. Recovery processes are executed in the control plane using a pre-trained machine learning model. Both traditional and advanced ML models, such as transformer and TimeGPT, are explicitly employed for data prediction. This approach ensures rapid and precise data recovery. Performance evaluations focus on recovery speed and accuracy, using a real dataset from a campus microgrid. With 20% missing PMU data, the mean absolute percentage error for voltage magnitude is 0.0384%, and the phase angle error discrepancy is approximately 0.4064%.

Phasor Measurement Unit, Machine Learning, Program↗
DOE OSTI2024

Ecological Benchmark for Chemicals

The Ecological Benchmark Tool dataset serves as a comprehensive repository of benchmarks designed to assess ecological risks at contaminated sites. This tool facilitates the evaluation of various environmental media and contaminants, supporting regulatory compliance and ecological protection. Benchmarks are available for air, biota, sediment, soil, and surface water. The dataset also provides species-specific benchmarks for fish, plants, birds, mammals, and invertebrates. Users can select benchmark sources, media, individual chemicals, and retrieve results in tabular or spreadsheet formats for analysis. Benchmarks are derived from authoritative sources, including government agencies, scientific councils, and academic publications. The dataset supports ecological risk assessments, regulatory decision-making, and environmental planning, with tools for benchmarking against chemical thresholds, sensitive species protection, and habitat impact evaluations. This structured approach ensures a robust evaluation of ecological risks tailored to site-specific and regulatory needs.

Stewart, Debra [Oak Ridge National Laboratory (ORN↗
DOE OSTI2026

Evaluating multi-slice ptychography tomography for X-ray imaging

X-ray ptychography typically relies on single-slice reconstruction approaches that assume minimal beam propagation through the specimen. However, this approximation breaks down for thick samples or when using high numerical aperture optics, leading to reconstruction artifacts due to multiple scattering. Multi-slice ptychographic tomography (MSPT) addresses this limitation by explicitly modeling beam propagation through the specimen while reducing angular sampling requirements for tomography, but its performance depends critically on reconstruction parameters and experimental conditions. Here, we systematically evaluate MSPT using numerical simulations and experimental X-ray datasets to understand how factors such as object thickness, optical parameters, and initialization strategies affect reconstruction accuracy. We demonstrate that informed object initialization significantly improves depth resolution and enables accurate projection extension, reducing the angular sampling requirements for tomographic reconstruction. Experimental demonstration confirms that MSPT produces reconstructions with reduced artifacts, establishing it as a robust approach for nanoscale imaging at modern synchrotron facilities.

Luktuke, Amey [Argonne National Laboratory (ANL), ↗
DOE OSTI2024

Understanding spatial and temporal drivers of variation in tree hydraulic processes and their consequences for climate feedbacks (Final Technical Report)

This is the final technical report from the first phase of a project that changed institutions. The grant was titled “Understanding spatial and temporal drivers of variation in tree hydraulic processes and their consequences for climate feedbacks.” The overall objectives of this project were to (1) provide model‐compatible datasets of key plant hydraulic traits and status for model evaluation, parameterization and validation and (2) use these data to pinpoint ecosystem responses to a changing hydroclimate by addressing both long‐term climatic drying and episodic extreme droughts. We planned to address the objectives with three research activities to quantify plant responses to chronic water stress and episodic drought: (1) generate high frequency observations of soil and plant hydraulic data across different landscape positions at multiple sites, (2) quantify plant hydraulic trait plasticity in response to experimental soil moisture reduction in situ in two central hardwood forests, and (3) simulate the carbon consequences of incorporating plant hydrodynamics and plant acclimation to water stress in the DOE‐sponsored plant hydrodynamics model FATES‐HYDRO. As of the transfer of this project to another institution, we had made substantial progress on activities 1 and 2, and started activity 3.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

TrojAI Alternate Analysis

In this portion of the TrojAI evaluation, we focus on the cyber-network-c2-mar2024 dataset. Recall that in this round ResNet18 and ResNet34 neural networks (NN) were trained on the USTC-TFC2016 dataset with the aim of distinguishing between benign versus botnet command and control (c2) packets. A range of bytes from each packet was reformatted into a 28x28 pixel image, and the collection of reformatted packets served as the training (and testing) data for the two ResNet models. For some of the data a trigger watermark was strategically placed to affect various inputs to the NNs. This watermarked, or poisoned, data in turn created a poisoned, or trojaned NN. The data were poisoned in different ways ultimately creating different trojaned NNs. This collection of trojaned NNs was combined with various versions of not trojaned NNs and served as the training and testing data for the performers. The performers’ task was to construct a classifier to distinguish between the trojaned and not trojaned models. It was previously noted that the performers struggled with the cyber-network-c2-mar2024 dataset, motivating this investigation of potential reasons the performers experienced challenges.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Model Card for WaveDenoiser

This study used STEAD to train and evaluate this model because STEAD is among the best benchmark datasets available for local to regional data. STEAD is a global dataset with over 1 million 60 s long seismic waveforms that originated from approximately 450,000 earthquakes and background noise captured by more than 2,500 seismic stations. Each waveform in STEAD was attached with metadata such as earthquake locations, station locations, and signal arrival times when available

58 GEOSCIENCES↗
DOE OSTI2024

Efficient Distributed Sequence Parallelism for Transformer-Based Image Segmentation

We introduce an efficient distributed sequence parallel approach for training transformer-based deep learning image segmentation models. The neural network models are comprised of a combination of a Vision Transformer encoder with a convolutional decoder to provide image segmentation mappings. The utility of the distributed sequence parallel approach is especially useful in cases where the tokenized embedding representation of image data are too large to fit into standard computing hardware memory. To demonstrate the performance and characteristics of our models trained in sequence parallel fashion compared to standard models, we evaluate our approach using a 3D MRI brain tumor segmentation dataset. We show that training with a sequence parallel approach can match standard sequential model training in terms of convergence. Furthermore, we show that our sequence parallel approach has the capability to support training of models that would not be possible on standard computing resources.

Lyngaas, Isaac↗
DOE OSTI2023

Data Validation Experiments with a Computer-Generated Imagery Dataset for International Nuclear Safeguards

Computer vision models have great potential as tools for international nuclear safeguards verification activities, but off-the-shelf models require fine-tuning through transfer learning to detect relevant objects. Because open-source examples of safeguards-relevant objects are rare, and to evaluate the potential of synthetic training data for computer vision, we present the Limbo dataset. Limbo includes both real and computer-generated images of uranium hexafluoride containers for training computer vision models. Here, we generated these images iteratively based on results from data validation experiments that are detailed here. The findings from these experiments are applicable both for the safeguards community and the broader community of computer vision research using synthetic data.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗