SEARCH · Engineering Papers

Results for “evaluation datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 343 records · Page 19

All sources NASA NTRS DOE OSTI

NASA NTRS2019

Improving Surface Soil Moisture Estimates in West Africa Through GRACE Data Assimilation

Recent annual trends of precipitation and terrestrial water storage (TWS) in West Africa have been increasing over the past decade. Despite a significant impact of soil moisture on the TWS in West Africa, there is little research on the recent spatial and temporal behaviors of surface soil moisture (SSM) along with the hydrological trends and variability in West Africa. In this study, we assimilate TWS estimates from the Gravity Recovery and Climate Experiment (GRACE) mission into the Catchment Land Surface Model (CLSM) and evaluate its impacts on SSM simulations for the years, 2002–2017. The evaluation is performed using reference datasets: the African Monsoon Multidisciplinary Analysis (AMMA) in situ soil moisture observations, three currently available microwave satellite SSM observations from the Advanced Scatterometer (ASCAT), the Soil Moisture Ocean Salinity (SMOS), and the Soil Moisture Active Passive (SMAP) satellites and also the triple collocation analysis (TCA). Overall, modeled SSM shows good agreement with the reference datasets in terms of the anomaly SSM correlations. However, both modeled and ASCAT SSM are limited in their representation of the drying rates, as observed by ground observations, SMOS and SMAP estimates. Further, GRACE data assimilation results in improved SSM simulations in the humid regions with large annual TWS variability. This study demonstrates the utility of land data assimilation to inform hydrological conditions in West Africa, where soil moisture monitoring is necessary for water resource and livestock management.

West Africa↗

NASA NTRS2021

A Three-Step Semi Analytical Algorithm (3SAA) for Estimating Inherent Optical Properties Over Oceanic, Coastal, and Inland Waters From Remote Sensing Reflectance

We present a three-step inverse model (3SAA) for estimating the inherent optical properties (IOPs) of surface waters from the remote sensing reflectance spectra, Rrs(). The derived IOPs include the total (a()), phytoplankton (aphy()), and colored detrital matter (acdm()), absorption coefficients, and the total (bb()) and particulate (bbp()) backscattering coefficients. The first step uses an improved neural network approach to estimate the diffuse attenuation coefficient of downwelling irradiance from Rrs. a() and bbp() are then estimated using the LS2 model (Loisel et al., 2018), which does not require spectral assumptions on IOPs and hence can assess a() and bb() at any wavelength at which Rrs() is measured. Then, an inverse optimization algorithm is combined with an optical water class (OWC) approach to assess aphy() and acdm() from anw().The proposed model is evaluated using an in situ dataset collected in open oceanic, coastal, and inland waters. Comparisons with other standard semi-analytical algorithms (QAA and GSM), as well as match-up exercises, have also been performed. The applicability of the algorithm on OLCI observations was assessed through the analysis of global IOPs spatial patterns derived from 3SAA and GSM. The good performance of 3SAA is manifested by median absolute percentage differences (MAPD) of 13%, 23%, 34% and 34% for bbp(443), anw(443), aphy(443) and acdm(443), respectively for oceanic waters. Due to the absence of spectral constraints on IOPs in the inversion of total IOPs, and the adoption of an OWC-based approach, the performance of 3SAA is only slightly degraded in bio-optical complex inland waters.

ocean color↗

NASA NTRSDate not supplied

NASA Investigation of Flow Direction Effects on Impedance Eduction for Acoustic Liners

In support of a collaboration with the International Forum for Aviation Research (IFAR), the NASA Langley Liner Physics Team has acquired a detailed dataset for use in the evaluation of flow direction effects on the acoustic liner impedance eduction process. Measurements are acquired with four acoustic liners designed to cover a range of sensitivities to source sound pressure level and tangential mean flow effects, specifically at source levels of 120 and 140 dB and centerline Mach numbers of 0.0, 0.1, 0.2, and 0.3. These measurements include detailed flow profile surveys upstream and downstream of the liner test window, as well as acoustic pressure data over the extent of the test window. The results clearly demonstrate the importance of the choice of Mach number used in the impedance eduction analysis. Average Mach number differences of less than 0.01 result in significant variation in the educed impedance spectra. These data will soon be supplied to the IFAR partners and will be made available for the general public within the next year.

acoustic↗

NASA NTRSDate not supplied

NASA Investigation of Flow Direction Effects on Impedance Eduction for Acoustic Liners

acoustic↗

NASA NTRS2024

Retrievals of Aerosol Optical Depth Over the Western North Atlantic Ocean During ACTIVATE

Aerosol optical depth was retrieved from two airborne remote sensing instruments, the Research Scanning Polarimeter (RSP) and Second Generation High Spectral Resolution Lidar (HSRL-2), during the National Aeronautics and Space Administration (NASA) Aerosol Cloud meTeorology Interactions oVer the western ATlantic Experiment (ACTIVATE). The field campaign offers a unique opportunity to evaluate an extensive 3-year dataset under a wide range of meteorological conditions from two instruments on the same platform. However, a long-standing issue in atmospheric field studies is that there is a lack of reference datasets for properly validating field measurements and estimating their uncertainties. Here we address this issue by using the triple collocation method, in which a third collocated satellite dataset from the Moderate Resolution Imaging Spectroradiometer (MODIS) is introduced for comparison. HSRL-2 is found to provide a more accurate retrieval than RSP over the study region. The error standard deviation of HSRL-2 with respect to the ground truth is 0.027. Moreover, this approach enables us to develop a simple, yet efficient, quality control criterion for RSP data. The physical reasons for the differences in two retrievals are determined to be cloud contamination, aerosols near the surface, multiple aerosol layers, absorbing aerosols, non-spherical aerosols, and simplified retrieval assumptions. These results demonstrate the pathway for optimal aerosol retrievals by combining information from both lidars and polarimeters for future airborne and satellite missions.

Aerosol optical depth↗

NASA NTRSDate not supplied

A Novel Framework for Multi-Path Data Fusion in Earth Observation and New Observing Strategies: Applications to Predicting Forest Canopy Height

Exponential growth of data from Earth Observation (EO) assets has necessitated the development of sophisticated methods for data interpretation and management. NASA’s New Observing Strategy (NOS) approach aims to coordinate operations among complex heterogenous systems of constellations, requiring advanced Artificial Intelligence and Machine Learning (AI/ML) techniques. Despite significant advancements in AI/ML across various domains, the EO and machine learning for satellite (SatML) fields remain fragmented, often relying on adapted techniques rather than domain-specific solutions. We present a novel end-to-end data fusion framework tailored specifically for EO and SatML, addressing this gap by facilitating rapid development of AI/ML applications. This framework, called, Multimodal Earth Observation Workflow for Machine Learning (MEOW-ML), sup- ports the entire AI/ML lifecycle, from dataset manipulation, to model training, evaluation, and logging, and is designed to expedite the development of next-generation NOS deployments and SOTA in EO. We apply our framework to predict canopy height model (CHM) derived from lidar data. We integrate multiple data modalities through a hierarchical, multi-path model architecture, effectively identifying and leveraging the unique strengths of each data source to enhance predictive accuracy. Our experiments demonstrate that the multi-path architecture outperforms traditional single-path models and provides significant advantages in both accuracy and computational efficiency.

Mark Moussa↗

NASA NTRSDate not supplied

Acoustic Testing of the Joby Aviation Propeller in the National Full-Scale Aerodynamics Complex 40- by 80-Foot Wind Tunnel

The advanced air mobility sector has progressed in recent years with electric vertical take-Off and landing vehicles at the forefront. In response, the NASA Advanced Air Mobility Project has initiated the Advanced Air Mobility National Campaign to partner with industry to progress this emerging market. This resulted in NASA’s Revolutionary Vertical Lift Technology project to participate to progress research efforts for software tool validation for these advanced air mobility vehicle configurations. Noise has been identified as one of the main obstacles to succeeding in this emerging market for community acceptance. To further understand electric vertical take-Off and landing vehicle noise, an isolated Joby Aviation S4 propeller was tested in the National Full-Scale Aerodynamics Complex 40- by 80-Foot Wind Tunnel to aid in providing high quality performance, loads, and acoustic data. Details of test hardware and setup are described with a focus on acoustics, which includes facility, test model, microphone locations, and data acquisition system details. Additionally, the data processing techniques and completed test matrix for acoustic points are provided. To aid in identifying potential acoustic reflections, an acoustic reflection test was performed prior to the start of the test and highlighted the presence of reflections at every microphone location. Acoustic results are provided for various flight regimes for sweeps of blade pitch, RPM, flow angle, and wind speed. These results show an optimum RPM for a specific thrust level for noise, and further reductions in RPM can increase noise levels. This dataset will be used to evaluate and improve computational tools.

Aviation Propeller↗

DOE OSTI2021

Fuzzy Simplicial Networks: A Topology-Inspired Model to Improve Task Generalization in Few-shot Learning

Deep learning has shown great success in settings with massive amounts of data but has struggled when data is limited. Few-shot learning algorithms, which seek to address this limitation, are designed to generalize well to new tasks with limited data. Typically, models are evaluated on unseen classes and datasets that are defined by the same fundamental task as they are trained for (e.g. category membership). One can also ask how well a model can generalize to fundamentally different tasks within a fixed dataset (for example: moving from category membership to tasks that involve detecting object orientation or quantity). To formalize this kind of shift we define a notion of “independence of tasks” and identify three new sets of labels for established computer vision datasets that test a model's ability to generalize to tasks which draw on orthogonal attributes in the data. We use these datasets to investigate the failure modes of metric-based few-shot models. Based on our findings, we introduce a new few-shot model called Fuzzy Simplicial Networks (FSN) which leverages a construction from topology to more flexibly represent each class from limited data. In particular, FSN models can not only form multiple representations for a given class but can also begin to capture the low-dimensional structure which characterizes class manifolds in the encoded space of deep networks. We show that FSN outperforms state-of-the-art models on the challenging tasks we introduce in this paper while remaining competitive on standard few-shot benchmarks.

deep learning↗

DOE OSTI2024

A Bayesian Learning Approach to Wireless Outdoor Heatmap Construction using Deep Gaussian Process

We present a novel Bayesian learning approach to outdoor radio heatmap construction utilizing deep Gaussian process (GP). The proposed approach employs a two-layer hierarchy which consists of two cascaded Gaussian processes that are capable of modeling more complex input-output relations than standard single-layer Gaussian processes. Since deriving the exact model likelihood is challenging, a lower bound is optimized instead so that gradient descent-based methods can be performed to find out the optimal model parameters. Typically, inducing points are used in GPs to facilitate low-rank approximation of covariance (kernel) matrices for computation speedup. However, the inaccuracy induced by inducing points can accumulate when stacking multiple layers of GP which may hinder the performance of deep GP. Moreover, since inducing points need to be learned, having them at all layers of deep GP also incurs computational burden. To overcome the above challenges, in contrast to the canonical deep GP model, we use a modified architecture where a full standard GP resides in the first layer and inducing points are only introduced for the second layer. This modified architecture strikes a balance between model accuracy and training complexity. In the proposed model, the noise parameter of the first GP layer is also eliminated to improve the training efficiency as the noise parameter at the output of the second layer suffices to model the uncertainty in the output. The proposed approach is evaluated on real-world datasets, in the form of location-Received Signal Strength (RSS) pairs, collected from the Platform for Open Wireless Data-driven Experimental Research (POWDER) located at the campus of the University of Utah. Experiment results show that the proposed approach can achieve smaller prediction errors on various training and testing data configurations than DNN-based and GP-based methods.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2023

Neural Image Compression: Generalization, Robustness, and Spectral Biases

Recent advances in neural image compression (NIC) have resulted in models which are starting to outperform traditional codecs. While this has led to growing excitement about using these methods in real-world applications, the successful adoption of any machine learning system (including NIC) in the wild requires it to generalize (and be robust) to unseen distribution shifts at deployment time. Unfortunately, current research lacks comprehensive datasets and informative tools to evaluate and understand compression performance in real-world settings. To bridge this crucial gap, first, this paper presents a comprehensive benchmark suite to evaluate the out-of-distribution (OOD) performance of image compression methods. Specifically, we design CLIC-C and Kodak-C by introducing 15 common corruptions to popular CLIC and Kodak benchmarks. Next, we propose spectrally inspired introspection tools to gain a deeper understanding of errors introduced by image compression methods as well as their OOD performance. To this end, we carry out a detailed performance comparison of the classical codec with various variants of NIC (e.g., original, variable rate, pruned), revealing intriguing findings that challenge our current understanding of the strengths and limitations of NIC. Finally, we corroborate our empirical findings with theoretical analysis, providing an in-depth view of the OOD performance of NIC. Our benchmarks, spectral introspection tools, and findings provide a crucial bridge to the real-world adoption of NIC. We hope that our work will propel future efforts in designing more robust and generalizable NIC methods.

neural networks, Variational Autoencoder, robustne↗

DOE OSTI2025

AI Benchmark Democratization and Carpentry

Benchmarks are a cornerstone of modern machine learning, enabling reproducibility, comparison, and scientific progress. However, AI benchmarks are increasingly complex, requiring dynamic, AI-focused workflows. Rapid evolution in model architectures, scale, datasets, and deployment contexts makes evaluation a moving target. Large language models often memorize static benchmarks, causing a gap between benchmark results and real-world performance. Beyond traditional static benchmarks, continuous adaptive benchmarking frameworks are needed to align scientific assessment with deployment risks. This calls for skills and education in AI Benchmark Carpentry. From our experience with MLCommons, educational initiatives, and programs like the DOE's Trillion Parameter Consortium, key barriers include high resource demands, limited access to specialized hardware, lack of benchmark design expertise, and uncertainty in relating results to application domains. Current benchmarks often emphasize peak performance on top-tier hardware, offering limited guidance for diverse, real-world scenarios. Benchmarking must become dynamic, incorporating evolving models, updated data, and heterogeneous platforms while maintaining transparency, reproducibility, and interpretability. Democratization requires both technical innovation and systematic education across levels, building sustained expertise in benchmark design and use. Benchmarks should support application-relevant comparisons, enabling informed, context-sensitive decisions. Dynamic, inclusive benchmarking will ensure evaluation keeps pace with AI evolution and supports responsible, reproducible, and accessible AI deployment. Community efforts can provide a foundation for AI Benchmark Carpentry.

von Laszewski, Gregor [Virginia U.]↗

DOE OSTI2025

Immersive Scientific Visualization of Molten-Salt Reactor Waste Characteristics Using Virtual Reality

Immersive visualization is changing how we explore, communicate, and understand complex scientific systems. In nuclear energy, an area in which data are often multidimensional, time-dependent, and difficult to interpret, virtual reality (VR) represents a powerful and intuitive informational medium. This work introduces a VR-based platform that visualizes the post-shutdown behavior and waste management lifecycle of molten-salt reactors (MSRs), a next-generation reactor type with unique operational and safety characteristics. The platform, built in Unity, is streamed on the Meta Quest 3 headset. It transforms high-fidelity simulation data into an interactive, immersive experience. Users can explore time-dependent reactor characteristics such as nuclide decay, which is a key factor for evaluating reactor waste strategies. The datasets were generated using the MOOSE (Multiphysics Object-Oriented Simulation Environment) framework and then processed through ParaView scripting for smooth integration into Unity. From a visualization standpoint, the platform emphasizes spatial storytelling, temporal exploration, and user-centered interaction. Users can navigate 3D reactor geometries, slice through volumetric data, and manipulate time to observe how physical phenomena evolve. Real-scale rendering and embodied interaction make the experience feel tangible. The interface is designed to be accessible, even to those without nuclear or simulation expertise. This lowers the barrier for stakeholders, policymakers, and the general public, while still supporting expert analysis and collaborative decision-making. This work shows how immersive visualization can function as both a scientific tool and a communication interface. By integrating simulation, processing, and visualization into a cohesive workflow, we offer a scalable framework for immersive scientific storytelling. The modular design supports future extensions to other reactor types and lifecycle stages, from shutdown to long-term storage, making the platform adaptable for both research and outreach.

99 - GENERAL AND MISCELLANEOUS↗

DOE OSTI2025

Performance Evaluation of Weather@home2 Simulations over West African Region

Weather and climate forecasting, using climate models, have become essential tools and life-savers in the West African region; in spite of the fact that climate models do not fully comply with attributes of forecast qualities—RASAP: reliability, association, skill, accuracy, and precision. The objective of this paper is to quantitatively evaluate, in comparison to CRU and ERA5 datasets, the RASAP compliance-level of the weather@home2 modeling system (w@h2). Findings from some statistical evaluations show that, to a moderately significant extent, w@h2 model provides useful information during the monsoon seasons; skills to capture the Little Dry Season over the Guinea zone; predictive skills for the onset season; ability to reproduce all the annual characteristics of the surface maximum air temperature over the region; as well as skill to detect heat waves that usually ravage West Africa during the boreal spring. The model displays traces of attributes that are needed for seasonal climate predictions and applications. Deficiencies in the quantitative reproducibility point to the facts that the model does provide a reliability akin to that of regional climate models. This paper further furnishes a prospective user with information on whether the model might be “useful or not” for a particular application.

West Africa↗

DOE OSTI2025

Next-Level Energy Management in Manufacturing: Facility-Level Energy Digital Twin Framework Based on Machine Learning and Automated Data Collection

This research introduces an energy prediction framework at the facility level supported by automated data collection and machine learning models. It investigates whether reducing the prediction time scale allows for applying more complex machine learning techniques and if those techniques improve the prediction accuracy. The primary advantages of this framework lie in its automation of the energy prediction process and its provision of real-time energy data suitable for use in energy dashboards or digital twins. A sitewide dataset was created by combining 15 min energy and daily production data of five shops—assembly, battery, body (electric), body (gas), and paint—from a globally recognized electric vehicle manufacturer. Various machine learning models were evaluated on daily, weekly, and monthly datasets, including, in increasingly complex order: naïve, simple linear regression, net regularized generalized linear regression, principal component regression, k-nearest neighbor, random forest, and Bayesian regularized neural network. Compared to the current state-of-the-art energy consumption prediction for the industrial facility level, this research investigates more complex models and smaller time intervals for higher accuracy. The findings revealed that the more complex monthly models require a minimum of a year and a half of data to operate, while weekly models demand a year of data to achieve improved accuracy. Daily models can operate with only six months of data but exhibit poor performance due to reduced prediction accuracy of production. Key challenges identified include access to reliable, high-quality energy and production data and the initial demand for human labor.

digital twin↗

DOE OSTI2025

An Evaluation of Representation Learning Methods in Particle Physics Foundation Models

We present a systematic evaluation of representation learning objectives for particle physics within a unified framework. Our study employs a shared transformer-based particle-cloud encoder with standardized preprocessing, matched sampling, and a consistent evaluation protocol on a jet classification dataset. We compare contrastive (supervised and self-supervised), masked particle modeling, and generative reconstruction objectives under a common training regimen. In addition, we introduce targeted supervised architectural modifications that achieve state-of-the-art performance on benchmark evaluations. This controlled comparison isolates the contributions of the learning objective, highlights their respective strengths and limitations, and provides reproducible baselines. We position this work as a reference point for the future development of foundation models in particle physics, enabling more transparent and robust progress across the community.

Chen, Michael [Caltech]↗

DOE OSTI2020

Assessing Current and Future Infrastructure Hazards

Project Objective Execute intelligent analytics via an advanced analytical framework, to assess the current state of offshore infrastructure, evaluate infrastructure life, and identify technologies to reduce infrastructure hazards, costs, and extend infrastructure life. Approach & Results thus Far• Build comprehensive dataset• Perform data-driven analytics to evaluate infrastructure integrity 1. Remaining lifespan 2. Likelihood of future risk• Apply data-driven advanced spatial, statistical, and Machine Learning (ML) models to quantify existing infrastructure integrity• Release data and models through a smart, online platform hosted by Energy Data eXchange (EDX)

Romeo, Lucy F.↗

DOE OSTI2020

Evaluation of historical CMIP6 model simulations of extreme precipitation over contiguous US regions

Simulated historical precipitation is evaluated for Coupled Model Intercomparison Project Phase 6 (CMIP6) models using precipitation indices defined by the Expert Team on Climate Change Detection and Indices. The model indices are evaluated against corresponding indices from the CPC unified gauge-based analyses of precipitation over seven geographical regions across the contiguous US (CONUS). The regions assessed match those in recent US National Climate Assessment Reports. To estimate observational uncertainty, precipitation indices for three other observational datasets (HadEx2, Livneh and PRISM) are evaluated against the CPC analyses. Both the moderate and extreme mean precipitation intensities are overestimated over the western CONUS and underestimated in the areas of the Central Great Plains (CGP) in most CMIP6 models tested. Most CMIP6 models overestimate the mean and variability of wet spell durations and underestimate the mean and variability of dry spell durations across the CONUS. Biases in interannual variability of most of the indices have similar patterns to those in corresponding mean biases. The median and interquartile model spreads in CMIP6 model biases are clearly smaller than those in CMIP5 model biases for wet spell durations. Multimodel medians of CMIP6 (CMIP6-MMM) and CMIP5 (CMIP5-MMM) have similar biases in climatology and variability but biases tend to be smaller in CMIP6-MMM. Depending on the index, extreme precipitation is slightly better in parts of the eastern half of the CONUS in CMIP6-MMM, otherwise, the biases in climatology and variability are similar to CMIP5-MMM. CMIP6-MMM performs better than individual models and even observational datasets in some cases. Differences between observational datasets for most indices are comparable to the CMIP6 interquartile model spread. The better-performing observational and model datasets are different in different parts of the CONUS.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

NASA NTRS2014

Review and Analysis of Algorithmic Approaches Developed for Prognostics on CMAPSS Dataset

Benchmarking of prognostic algorithms has been challenging due to limited availability of common datasets suitable for prognostics. In an attempt to alleviate this problem several benchmarking datasets have been collected by NASA's prognostic center of excellence and made available to the Prognostics and Health Management (PHM) community to allow evaluation and comparison of prognostics algorithms. Among those datasets are five C-MAPSS datasets that have been extremely popular due to their unique characteristics making them suitable for prognostics. The C-MAPSS datasets pose several challenges that have been tackled by different methods in the PHM literature. In particular, management of high variability due to sensor noise, effects of operating conditions, and presence of multiple simultaneous fault modes are some factors that have great impact on the generalization capabilities of prognostics algorithms. More than 70 publications have used the C-MAPSS datasets for developing data-driven prognostic algorithms. The C-MAPSS datasets are also shown to be well-suited for development of new machine learning and pattern recognition tools for several key preprocessing steps such as feature extraction and selection, failure mode assessment, operating conditions assessment, health status estimation, uncertainty management, and prognostics performance evaluation. This paper summarizes a comprehensive literature review of publications using C-MAPSS datasets and provides guidelines and references to further usage of these datasets in a manner that allows clear and consistent comparison between different approaches.

Uncertainty↗