SEARCH · Engineering Papers

Results for “evaluation datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 739 records · Page 41

All sources NASA NTRS DOE OSTI

DOE OSTI2021

Deep-Learning-Derived Evaluation Metrics Enable Effective Benchmarking of Computational Tools for Phosphopeptide Identification

Tandem mass spectrometry (MS/MS)-based phosphoproteomics is a powerful technology for global phosphorylation analysis. However, applying four computational pipelines to a typical mass spectrometry (MS)-based phosphoproteomic dataset from a human cancer study, we observed a large discrepancy among the reported phosphopeptide identification and phosphosite localization results, underscoring a critical need for benchmarking. While efforts have been made to compare performance of computational pipelines using data from synthetic phosphopeptides, evaluations involving real application data have been largely limited to comparing the numbers of phosphopeptide identifications due to the lack of appropriate evaluation metrics. We investigated three deep learning-derived features as potential evaluation metrics: phosphosite probability, Delta RT and spectral similarity. Predicted phosphosite probability is computed by MusiteDeep, which provides high accuracy as previously reported; Delta RT is defined as the absolute retention time (RT) difference between RTs observed and predicted by AutoRT; and spectral similarity is defined as the Pearson’s correlation coefficient between spectra observed and predicted by pDeep2. Using a synthetic peptide dataset, we found that both Delta RT and spectral similarity provided excellent discrimination between correct and incorrect peptide-spectrum matches (PSMs) both when incorrect PSMs involved wrong peptide sequences and even when incorrect PSMs were caused by only incorrect phosphosite localization. Based on these results, we used all the three deep learning-derived features as evaluation metrics to compare different computational pipelines on diverse set of phosphoproteomic datasets and showed their utility in benchmarking performance of the pipelines. The benchmark metrics demonstrated in this study will enable users to select computational pipelines and parameters for routine analysis of phosphoproteomics data and will offer guidance for developers to improve computational methods.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2026

Timeseries Photos of a Variably Inundated Stream: Umtanum Creek, Washington, United States

This dataset is associated with a broader study using game camera timeseries photos collected to evaluate stream variable inundation via changes in width (i.e. wet fraction). Four game cameras were deployed along Umtanum Creek (Washington, United States) to track changes in stream inundation over time. Drone imagery was collected at the same location on October 18, 2024 which was used to construct a digital elevation model (DEM) of the streambed topography. The associated paper and data can be found at https://doi.org/10.1016/j.envsoft.2025.106715 (Bao et al., 2025a)) and https://doi.org/10.15485/2589885 (Bao et al., 2025b), respectively. For details on how to navigate data packages generated by this project, see https://data.ess-dive.lbl.gov/portals/PNNLRiverCorridorSFA/About. In addition to this readme, this data package also includes a file-level metadata (FLMD) files that describes each file and a data dictionaries (DD) that describe all column/row headers and variable definitions. This dataset is comprised of (1) file-level metadata; (2) data dictionary; (3) readme; (4) field metadata; (5) field protocol; and (5) folders containing game camera photos. Game camera photos are organized into folders for each camera (CDL, CUL, CDR, CUR; see readme for information on camera naming) by the month photos were collected. All files are .csv, .jpg, or .pdf.

AI image segmentation↗

DOE OSTI2023

Identifying strengths and weaknesses of methods for computational network inference from single-cell RNA-seq data

Single-cell RNA-sequencing (scRNA-seq) offers unparalleled insight into the transcriptional programs of different cellular states by measuring the transcriptome of thousands of individual cells. An emerging problem in the analysis of scRNA-seq is the inference of transcriptional gene regulatory networks and a number of methods with different learning frameworks have been developed to address this problem. Here, we present an expanded benchmarking study of eleven recent network inference methods on seven published scRNA-seq datasets in human, mouse, and yeast considering different types of gold standard networks and evaluation metrics. We evaluate methods based on their computing requirements as well as on their ability to recover the network structure. We find that, while most methods have a modest recovery of experimentally derived interactions based on global metrics such as Area Under the Precision Recall curve, methods are able to capture targets of regulators that are relevant to the system under study. Among the top performing methods that use only expression were SCENIC, PIDC, MERLIN or Correlation. Addition of prior biological knowledge and the estimation of transcription factor activities resulted in the best overall performance with the Inferelator and MERLIN methods that use prior knowledge outperforming methods that use expression alone. We found that imputation for network inference did not improve network inference accuracy and could be detrimental. Comparisons of inferred networks for comparable bulk conditions showed that the networks inferred from scRNA-seq datasets are often better or at par with the networks inferred from bulk datasets. Our analysis should be beneficial in selecting methods for network inference. At the same time, this highlights the need for improved methods and better gold standards for regulatory network inference from scRNAseq datasets.

59 BASIC BIOLOGICAL SCIENCES↗

NASA NTRS2021

Evaluation of Clouds, Radiation, and Precipitation in CMIP6 Models Using Global Weather States Derived from ISCCP-H Cloud Property Data

A clustering methodology is applied to cloud optical depth cloud top pressure (TAU-PC) histograms from the new, 1-degree resolution, ISCCP-H dataset, to derive an updated global Weather State (WS) dataset. Then, PC-TAU histograms from current-climate CMIP6 model simulations are assigned to the ISCCP-H WSs along with their concurrent radiation and precipitation properties, to evaluate model cloud, radiation, and precipitation properties in the context of the Weather States. The new ISCCP-H analysis produces WSs that are very similar to those previously found in the lower resolution ISCCP-D dataset. The main difference lies in the splitting of the ISCCP-D thin stratocumulus WS between the ISCCP-H shallow cumulus and stratocumulus WSs, which results in the reduction by one of the total WS number. The evaluation of the CMIP6 models against the ISCCP-H Weather States, shows that, in the ensemble mean, the models are producing an adequate representation of the frequency and geographical distribution of the WSs, with measurable improvements compared to the WSs derived for the CMIP5 ensemble. However, the frequency of shallow cumulus clouds continues to be underestimated, and, in some WSs the good agreement of the ensemble mean with observations comes from averaging models that significantly overpredict and underpredict the ISCCP-H WS frequency. In addition, significant biases exist in the internal cloud properties of the model WSs, such as the model underestimation of cloud fraction in middle-top clouds and secondarily in midlatitude storm and stratocumulus clouds, that result in an underestimation of cloud SW cooling in those regimes.

Clouds↗

DOE OSTI2022

Disentangling error structures of precipitation datasets using decision trees

Characterizing error structures in precipitation products not only facilitates their proper applications for scientific and practical purposes but also helps improve their retrieval algorithms and processing methods. Despite the fact that multiple precipitation products have been assessed in the literature, factors that affect their error structures remain inadequately addressed. By interpreting 60 binary decision trees, this study disentangles the error characteristics of precipitation products in terms of their spatiotemporal patterns and geographical factors. Three independent precipitation products - two satellite-based and one reanalysis datasets: the Integrated Multi-satellitE Retrievals for GPM (Global Precipitation Measurement) late run (IMERG-L), Soil Moisture to Rain-Advanced SCATterometer (SM2RAIN-ASCAT), and the Modern-Era Retrospective analysis for Research and Applications, Version 2 uncorrected precipitation output (MERRA2-UC), are evaluated across the contiguous United States from 2010 to 2019. Here, the ground-based Stage IV precipitation dataset is used as the ground truth. Results indicate that the MERRA2-UC outperforms the IMERG-L and SM2RAIN-ASCAT with higher accuracy and more stable interannual patterns for the analysis period. Decision trees cross-assess three spatiotemporal factors and find that the underestimation of MERRA2-UC occurs in the east of the Rocky Mountains, and SM2RAIN-ASCAT underestimates precipitation over high latitudes, especially in winter. Additionally, the decision tree method ascribes system errors to nine different geographical characteristics, of which the distance to the coast, soil type, and DEM are the three dominant features. On the other hand, the land cover type, topography position index, and aspect are three relatively weak factors.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Single reader between-cases AUC estimator with nested data

The area under the receiver operating characteristic curve (AUC) is widely used in evaluating diagnostic performance for many clinical tasks. It is still challenging to evaluate the reading performance of distinguishing between positive and negative regions of interest (ROIs) in the nested-data problem, where multiple ROIs are nested within the cases. To address this issue, we identify two kinds of AUC estimators, within-cases AUC and between-cases AUC. We focus on the between-cases AUC estimator, since our main research interest is in patient-level diagnostic performance rather than location-level performance (the ability to separate ROIs with and without disease within each patient). Another reason is that as the case number increases, the number of between-cases paired ROIs is much larger than the number of within-cases ROIs. We provide estimators for the variance of the between-cases AUC and for the covariance when there are two readers. We derive and prove the above estimators’ theoretical values based on a simulation model and characterize their behavior using Monte Carlo simulation results. We also provide a real-data example. Moreover, we connect the distribution-based simulation model with the simulation model based on the linear mixed-effect model, which helps better understand the sources of variation in the simulated dataset.

Health Care Sciences & Services↗

DOE OSTI2024

Evaluation of Global Climate Models for Use in Energy Analysis

The interplay between energy, climate, and weather is becoming more complex due to increasing contributions of renewable energy generation, energy storage, electrified end uses, and the increasing frequency of extreme weather events. Energy system analyses commonly rely on meteorological inputs to estimate renewable energy generation and energy demand; however, these inputs rarely represent the estimated impacts of future climate change. Climate models and publicly available climate change datasets can be used for this purpose, but the selection of inputs from the myriad of available models and datasets is a nuanced and subjective process. In this work, we assess datasets from various global climate models (GCMs) from the Coupled Model Intercomparison Project Phase 6 (CMIP6). We present evaluations of their skills with respect to the historical climate and comparisons of their future projections of climate change for two climate change scenarios. We present the results for different climatic and energy system regions and include interactive figures in the accompanying software repository. Previous work has presented similar GCM evaluations, but none have presented variables and metrics specifically intended for comprehensive energy systems analysis including impacts on energy demand, thermal cooling, hydropower, water availability, solar energy generation, and wind energy generation. We focus on GCM output meteorological variables that directly affect these energy system components including the representation of extreme values that can drive grid resilience events. The objective of this work is not to recommend the best climate model and dataset for a given analysis, but instead to provide a reference to facilitate the selection of climate models and scenarios in subsequent work.

14 SOLAR ENERGY↗

DOE OSTI2026

Data & Code from Phoenix CPPP Phase 2 Analysis

This data and code package supports the analysis presented in “Beyond Surface Cooling: Comprehensive Field Assessment of Reflective Pavement Thermal Performance in Phoenix, Arizona” and provides fully reproducible workflows for evaluating the thermal performance of cool pavement treatments in a hot urban environment. The dataset integrates multi-modal field measurements collected across residential and nonresidential settings, including mobile air temperature traverses, stationary air temperature monitoring, residential mean radiant temperature (MRT) measurements, subsurface temperature profiles, and controlled testbed observations. The data package contains raw and processed datasets in comma-separated value (CSV) format, accompanying metadata files describing site characteristics and measurement protocols, and R scripts (.R files) used for data cleaning, time synchronization, spatial and temporal matching, quality control filtering, statistical comparison, and figure generation. All analyses were conducted using R (version ≥ 4.2.0) with commonly available packages (e.g., tidyverse, lubridate, data.table, ggplot2). No proprietary software is required to reproduce results. Field campaigns were designed to quantify the effects of high-reflectance pavement coatings on surface temperature, near-surface air temperature, subsurface heat propagation, and radiative heat exposure. Temporal alignment procedures include standardized timestamp conversion and nearest-neighbor matching of high-frequency sensor measurements to stop-based metadata within defined tolerance windows to ensure comparability across instruments. The workflows generate summary statistics, treatment–control contrasts, depth-dependent thermal gradients, and time-series visualizations used in the associated publication. By integrating mobile, stationary, radiative, and subsurface measurements within a unified and transparent processing framework, this package enables comprehensive evaluation of cool pavement performance across multiple thermal exposure pathways and supports reuse in future urban heat mitigation and climate resilience studies.

AIR TEMPERATURE↗

DOE OSTI2023

Low-Temperature Geothermal Geospatial Datasets: An Example from Alaska: Preprint

This project is part of a broader effort focused on geothermal heating and cooling with the purpose of demonstrating the multi-faceted value of integrating GHC/GHX into national decarbonization plans and community energy plans. Currently, there is a need to better define and evaluate low-temperature geothermal resource potential to provide the basis for supporting private sector investment. For shallow GHC/GHX fields, there is no formal compilation of subsurface characteristics shared among industry practitioners that can improve system design and operations. Alaska is specifically noted in this work, because heretofore, it has not received a similar focus in geothermal potential evaluations as the contiguous United States. The methodology consists of leveraging relevant data generate a baseline low-temperature resources (< 150 degrees C) geospatial dataset to compare and analyze information accessible to anyone trying to understand the potential of GHC/GHX and small-scale low-temperature geothermal power in Alaska (e.g., energy modelers, communities, planners, and policymakers). Importantly, this project is identifying data related to (1) the evaluation of GHC/GHX in the shallow subsurface, and (2) the evaluation of low-temperature geothermal resource availability. Additionally, data is being compiled to assess repurposing of oil and gas wells to contribute co-produced fluids toward the geothermal direct use and heating and cooling resource potential. The goal of this project is facilitating future deployment of GHC/GHX analysis and community-led programs and update the low-temperature geothermal resources assessment of Alaska. A better understanding of shallow potential for geothermal heat exchange (GHX) will improve design and operations of highly efficient GHC systems. The deployment and impact that can be achieved for low temperature geothermal resources will contribute to decarbonization goals and facilitate widespread electrification by shaving and shifting grid loads.

bottom-hole temperature↗

NASA NTRSDate not supplied

Development of a Global Reference Surface Reflectance and BRDF Datasets from Geostationary Satellite Observations and AERONET Measurements

Surface reflectances and their dependency on illumination-view geometries (i.e., BRDF) are the foundation of many high-level satellite products for land and water monitoring. Yet it is difficult to evaluate the quality of satellite-based surface reflectances with ground-based measurements due to the spatial scale differences. In order to fill the gap, here we develop a reference dataset of surface reflectance and BRDF at the global AERONET sites with data streams from operational geostationary sensors including Himawari 8/9 AHI, GK-2A AMI, and GOES 16/17 ABI. Taking the top-of-atmosphere (TOA) reflectance and the site measured atmospheric aerosol optical depth (AOD) as the main inputs, we apply the GeoNEX-AC algorithm to performance accurate atmospheric correction and derive 10-minute surface reflectance and daily Ross-Thick-Li-Sparse (RTLS) BRDF parameters at AERONET sites where coincident AOD measurements and TOA observations are available from 2016 (for Himawari) or 2018 (for GOES) onwards. The algorithm ensures that the retrieved surface BRDF parameters, along with the site-measured AOD, allow the atmospheric radiative transfer model, SHARM, accurately simulate the observed TOA reflectance at diurnal and longer time scales. They are our best estimates of the surface optical properties and thus can serve as the “reference” to evaluate the performance of operational atmospheric correction algorithms (where AOD is assumed unknown and needs to be retrieved). The reference BRDF also allow us to evaluate the spectral band ratios between the SWIR (e.g., 2200 nm) and the visible (e.g., 650 nm) regions, which are commonly used in operational atmospheric correction algorithms. Finally, we demonstrate that the reference dataset can be used to develop potential data synergies between different GEO satellites as well as GEO-LEO sensors.

Weile Wang↗

DOE OSTI2023

GEESS as a Mechanism to Facilitate the Commercialization of Geologic Carbon Sequestration

It is a presentation on the Geoanalytical Economic Evaluation of Saline Storage (GEESS) project. The GEESS project objectives are to characterize in detail geologic saline formations targeted for geologic carbon sequestration (GCS) using publicly available datasets and create high spatial resolution datasets (up to 5 km grid) of geologic parameters. The presentation aims to provide a comprehensive list of references used to characterize each geologic formation evaluated so far.

Eppink, Jeffrey↗

DOE OSTI2023

Projected changes in extreme rainfall and temperature events and possible implications for Cameroon's socio-economic sectors

Extreme events like flooding, droughts and heatwave are among the factors causing huge socio-economic losses to Cameroonians. Investigating the potential response of rainfall and temperature extremes to global warming is therefore critically needed for tailoring and adjusting the country's policies. Recent datasets have been developed for this purpose within the Coordinated Output for Regional Evaluations (CORDEX-CORE) initiative, at ~25 km grid spacing. These regional climate models were used to dynamically downscaled four global climate models participating in the Coupled Model Intercomparison Project phase 5 (CMIP5), under the optimistic and pessimistic representative concentration pathways (RCPs) 2.6 and 8.5, respectively. These models were employed in this study for characterizing the response of Cameroon's extreme precipitation and temperature events to global warming, using seven indices defined by the Expert Team on Cli mate Change Detection and Indices. Under global warming, the maximum number of consecutive dry (wet) days' is expected to increase (decrease). However, the annual total rainfall amount is expected to increase, mainly due to the intensification of very wet days and daily rainfall intensity. Furthermore, the temperature-based indices reveal an increase (decrease) in the total annual hot (cold) days, and overall, changes intensify with increased radiative forcing. The high-mitigated low-emission pathway RCP2.6 features attenuated changes, and even sometimes adapts to reverse the sign of changes. Designing reliable policies to limit the risks associated with the above changes is required, as their socio-economic consequences are likely to include food insecurity, heat-related illness, population impoverishment, price rises and market instability.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2021

Lessons Learned in Employing Data Analytics to Predict Oxidation Kinetics and Spallation Behavior of High-Temperature NiCr-Based Alloys

Machine learning (ML) can offer many advantages in predicting material properties over traditional materials development methods based solely on limited experimental investigations or physical-based simulations with the capability to reduce development cost, risk, and time. However, so far, limited efforts have been made to predict alloy oxidation kinetics and spallation behavior via ML due to the lack of consistently measured and sufficient experimental data and the inherent complexity in oxidation behavior of multicomponent high-temperature alloys. A previous study reported the ability of ML to predict oxidation kinetics of NiCr-based alloys as a function of alloy composition and operating conditions. Here, the performance of a ML model in predicting rate constants and spallation probability was evaluated in light of the roles of the data distribution of the experimental dataset (data analytics), the alloy composition, the exposure environment and the chosen oxidation approach to extracting kinetic values from the measured mass changes (but using either a simple parabolic law or a statistical cyclic oxidation model). Potential strategies to improve the predictions and enhance the extrapolative capability of the previously trained model will be discussed.

36 MATERIALS SCIENCE↗

DOE OSTI2025

Generalizable machine learning potentials for quantum-accurate predictions of non-equilibrium behavior in 2D materials

Machine learning interatomic potentials (ML-IAPs) are emerging as transformative tools in materials modeling, promising quantum-level accuracy at a fraction of the computational cost. However, their ability to generalize beyond equilibrium configurations and to reliably capture defect- and temperature-driven behavior remains underexplored. Here, we develop and benchmark two state-of-the-art ML-IAPs, Spectral Neighbor Analysis Potential (SNAP) and Allegro, on a comprehensive dataset for monolayer MoSe₂. Using density functional theory (DFT) as the reference, we evaluate their performance in capturing stress–strain behavior, phase transition energetics, defect evolution, edge stability, and fracture toughness. Allegro, a deep equivariant neural network potential, surpasses both SNAP and the classical Tersoff potential in accuracy, efficiency, and transferability. Importantly, both ML potentials accurately reproduce experimental fracture measurements and ab initio predictions of inversion domain formation—phenomena well beyond their training sets. Our findings establish ML-IAPs as viable replacements for traditional force fields in the study of non-equilibrium mechanical phenomena, enabling large-scale, high-fidelity simulations in 2D materials and beyond. In conclusion, this work provides a broadly applicable framework for data-driven modeling of structural and functional transformations under extreme conditions.

2D materials↗

DOE OSTI2022

Non-destructive measurement and real-time monitoring of apple hardness during ultrasonic contact drying via portable NIR spectroscopy and machine learning

Portable near-infrared spectrometer in the spectral range of 900–1700 nm was evaluated for the first time to assess and monitor apple hardness in real-time during ultrasonic drying. Calibration models were developed using PLS and ANN, and their performances were evaluated by internal leave-one-out cross-validation and an external dataset. Several pre-treatments including standard normal variate (SNV), multiplicative scatter correction (MSC), Savitzky–Golay first and second derivatives were employed to examine the effects of spectral variations in hardness prediction. Seven important wavelengths were selected using weighted regression coefficients to develop a simple MLR model to facilitate the model interpretation and circumvent noise. The models using PLS, MLR, and ANN with selected wavelengths predicted the apple hardness with R 2 p of 0.91, 0.91, 0.95, and RMSEP of 14.78, 14.85, and 12.46 N, respectively. Finally, the results indicate that portable NIR spectrometers are quite promising for real-time monitoring of apple hardness during ultrasonic drying.

47 OTHER INSTRUMENTATION↗

DOE OSTI2024

Evaluating direct vessel injection accident-event progression of AP1000 and key figures of merit to support the design and development of water-cooled small modular reactors

The passive safety systems (PSSs) within water-cooled reactors are meticulously engineered to function autonomously, requiring no external power source or manual intervention. They depend exclusively on inherent natural forces and the fundamental principles of reactor physics, such as gravity, natural convection, and phase changes, to manage, alleviate, and avert the release of radioactive materials into the environment during accident scenarios like a loss-of-coolant accident (LOCA). PSSs are already integrated into such operating commercial reactors as the Advanced Pressurized Reactor-1000 MWe (AP1000) and the Water-Water Energetic Reactor-1200 MWe (WWER-1200) are adopted in most of the upcoming small modular reactor (SMR) designs. Examples of water-cooled SMR PSSs are the passive emergency core-cooling system (ECCS), passive containment cooling system (PCCS), and passive decay-heat removal system, the designs of which vary based on reactor system-design requirements. However, understanding the accident-event progression and phases of a LOCA is pivotal for adopting a specific PSS for a new SMR design. This study covers the accident-event progression for direct vessel injection (DVI) small-break loss-of-coolant accident (SB-LOCA), associated physics phenomena, knowledge gaps, and important figures of merit (FOMs) that may need to be evaluated and assessed to validate thermal-hydraulics models with an available experimental dataset to support new SMR design and development.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

DOE OSTI2021

Active and machine learning-based approaches to rapidly enhance microbial chemical production

In order to make renewable fuels and chemicals from microbes, new methods are required to engineer microbes more intelligently. Computational approaches, to engineer strains for enhanced chemical production typically rely on detailed mechanistic models (e.g., kinetic/stoichiometric models of metabolism)—requiring many experimental datasets for their parameterization—while experimental methods may require screening large mutant libraries to explore the design space for the few mutants with desired behaviors. To address these limitations, we developed an active and machine learning approach (ActiveOpt) to intelligently guide experiments to arrive at an optimal phenotype with minimal measured datasets. In this study, ActiveOpt was applied to two separate case studies to evaluate its potential to increase valine yields and neurosporene productivity in Escherichia coli. In both the cases, ActiveOpt identified the best performing strain in fewer experiments than the case studies used. This work demonstrates that machine and active learning approaches have the potential to greatly facilitate metabolic engineering efforts to rapidly achieve its objectives.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2022

Classifying handedness in chiral nanomaterials using label error robust deep learning

Abstract High-throughput scanning electron microscopy (SEM) coupled with classification using neural networks is an ideal method to determine the morphological handedness of large populations of chiral nanoparticles. Automated labeling removes the time-consuming manual labeling of training data, but introduces label error, and subsequently classification error in the trained neural network. Here, we evaluate methods to minimize classification error when training from automated labels of SEM datasets of chiral Tellurium nanoparticles. Using the mirror relationship between images of opposite handed particles, we artificially create populations of varying label error. We analyze the impact of label error rate and training method on the classification error of neural networks on an ideal dataset and on a practical dataset. Of the three training methods considered, we find that a pretraining approach yields the most accurate results across label error rates on ideal datasets, where size and other morphological variables are held constant, but that a co-teaching approach performs the best in practical application.

36 MATERIALS SCIENCE↗