Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “validation dataset”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 91 records · Page 5
All sourcesNASA NTRSDOE OSTI
NASA NTRS2023

Seasonal Forecasting Skill for the High Mountain Asia Region in the Goddard Earth Observing System

Seasonal variability of the global hydrologic cycle directly impacts human activities, including hazard assessment and mitigation, agricultural decisions, and water resources management. This is particularly true across the High Mountain Asia (HMA) region, where availability of water resources can change depending on local seasonality of the hydrologic cycle. Forecasting the atmospheric states and surface conditions, including hydrometeorological relevant variables, at subseasonal-to-seasonal (S2S) lead times of weeks-to-months is an area of active research and development. NASA’s 15 Goddard Earth Observing System (GEOS) S2S prediction system has been developed with this research goal in mind. Here, we benchmark the forecast skill of GEOS-S2S (version 2) hydrometeorological forecasts at 1-3 month lead times in the HMA region, including a portion of the Indian Subcontinent, during the retrospective forecast period, 1981-2016. To assess forecast skill, we evaluate 2-m air temperature, total precipitation, fractional snow cover, snow water equivalent, surface soil moisture, and terrestrial water storage forecasts against the Modern-Era Retrospective analysis for Research and 20 Applications, Version 2 (MERRA-2) and independent reanalysis data, satellite observations, and data fusion products. Anomaly correlation is highest when the forecasts are evaluated against MERRA-2 and particularly in variables with long memory in the climate system, likely due to similar initial conditions and model architecture used in GEOS-S2S and MERRA-2. When compared to MERRA-2, results for the 1-month forecast skill range from anomaly correlation of R anom =0.18 for precipitation to R anom =0.62 for soil moisture. Anomaly correlations are consistently lower when forecasts are 25 evaluated against independent observations; results for the 1-month forecast skill range from R anom =0.13 for snow water equivalent to R anom =0.24 for fractional snow cover. We find that, generally, hydrometeorological forecast skill is dependent on the forecast lead time, the memory of the variable within the physical system, and the validation dataset used. Overall, these results benchmark the GEOS-S2S system’s ability to forecast HMA hydrometeorology.

GEOS↗
NASA NTRSDate not supplied

Updates on the DebriSat Hypervelocity Experiment and Characterization of Fragments in Support of Environmental Models

To develop, maintain, and update orbital debris environmental and break-up models, such as the NASA Orbital Debris Engineering Model (ORDEM) and the NASA Standard Satellite Breakup Model (SSBM), the NASA Orbital Debris Program Office (ODPO) relies on the most recent validated datasets from various measurement sources, both laboratory and environmental. One key project that will provide insight for break-up events using modern-day spacecraft materials and construction techniques is the DebriSat laboratory hypervelocity-impact test. Based on the mass of the target, the projectile, and the impact velocity, the expected number of fragments greater than 2 mm, generated using the NASA SSBM, was estimated to be close to 85,000. To date, the DebriSat fragment database continues to grow, with over 200,000 fragments collected that will help inform updates to the SSBM. Additionally, the growing fragment ensemble will support key parameters for the next release of NASA’s environmental models, employing fragment shapes, densities, and size distributions. To further support these environmental models, specifically the size parameter used in ground-based optical measurements, optical characterization on a subset of DebriSat fragments is being conducted in ODPO’s Optical Measurement Center (OMC). Broadband bidirectional reflectance distribution (BRDF) measurements will provide insight into the optical-based NASA Size Estimation Model (OSEM). The OSEM equates an object’s brightness to size (e.g., diameter of a disk or sphere) given several assumed parameters, including a defined phase function, albedo, and range. To address the first defined parameter, the ODPO has been using ray-tracing software to simulate light conditions in the OMC and to generate phase functions (i.e., specular, Lambertian, and experimentally-derived) of known shapes, materials, and sizes. This simulated data, in addition to the experimentally derived measurements collected in the OMC, will aid in determining whether a new phase function would be suitable for an updated OSEM. The OSEM also assumes a single-value albedo, thus preimpact spectral measurements on a subset of DebriSat materials were acquired for baseline material characterization and to provide insight into spacecraft material taxonomies. This DebriSat spectral data, along with spectral measurements from other known spacecraft material samples, will allow for better analyses of albedo variations and the effect on size calculations of defined laboratory targets, thus further supporting OSEM updates. This paper provides an overview of DebriSat, the status of the project, updates on the parameter distributions, an overview of the NASA SSBM, and ongoing fragment characterization efforts within the OMC.

Heather Cowardin↗
NASA NTRSDate not supplied

Evaluating Meteorological Dust Events and Machine-Learning Based Dust Identification in Geostationary Satellite Imagery

NASA scientists in the Short-term Prediction Research and Transition Center (SPoRT) developed a physically-based machine learning approach to identify dust in satellite imagery with a focus on night-time dust detection (Berndt et al. 201; DustTracker-AI). NASA/NOAA Geostationary Environmental Operational Satellite-16 (GOES-16) imagery was used for training and model inputs. The training, testing and validation data set consists of 28 events in the Southwest United States, capturing dust and null events in the region from 2018-2020.With 83 distinct images and millions of pixels a random forest model was trained and validated, correctly labeling 85% of dust pixels.For the first time, the model was run in near-real time production during the spring of 2022 and dust probability visualizations were made available to NOAA National Weather Service (NWS) forecasters to assess its utility for dust forecasting. Results indicated the model helped increase the confidence in the presence of dust and enabled dust tracking for a longer period of time into the night-time hours. Forecaster assessment and running the model in near real-time allowed for the team to determine the types of events missed, captured, and false alarms. To gain additional context on model performance,the SPoRT team sought to gather more detailed information on the training database(e.g., meteorological characteristics and drivers). The goal of this project was to identify the meteorological drivers for the dust events and create a database which synthesized information from observations, forecaster discussions, and analyses pertaining to the dust events to understand the types of events currently used to train the model. A more detailed meteorological synopsis was created for each dust event in the training, testing, and validation datasets. Following the completion of the database and documentation, the classification details revealed that 88% of the dust events were synoptically driven while mesoscale events were less prevalent in model datasets. Meteorological conditions found such as mixing layer depth and wind velocity had mean values of 645mb and 21kt respectively.With conditions of deep mixed layers and moderate to strong surface winds a mesoscale thunderstorm outflow event was considered and subsequently added to the model training data set to test the impact of additional mesoscale training data. The model was retrained and then qualitatively tested on a sample thunderstorm outflow case that the original model was unable to identify. Preliminary results showed potential that the addition of more mesoscale events included in the training data could help to better identify indistinct and localized dust events.

Connor Welch↗
NASA NTRS2024

Summarizing Multiple Aspects of Triple Collocation Analysis in a Single Diagram

With the ongoing expansion of global observation networks, it is expected that we shall routinely analyze records of geophysical variables such as temperature from multiple collocated instruments. Validating datasets in this situation is not a trivial task because every observing system has its own bias and noise. Triple collocation is a general statistical framework to estimate the error characteristics in three or more observational-based datasets. In a triple colocation analysis, several metrics are routinely reported but traditional multiple-panel plots are not the most effective way to display information. A new formula of error variance is derived for connecting the key terms in the triple collocation theory. A diagram based on this formula is devised to facilitate triple collocation analysis of any data from observations, as illustrated using three aerosol optical depth datasets from the recent Aerosol Cloud meTeorology Interactions oVer the western ATlantic Experiment (ACTIVATE). An observational-based skill score is also derived to evaluate the quality of three datasets by taking into account both error variance and correlation coefficient. Several applications are discussed and sample plotting routines are provided.

triple collocation↗
NASA NTRSDate not supplied

Establishing an in-Space Joining Ecosystem at NASA Marshall via Laser Beam Welding

NASA Marshall is establishing an ISAM technology development ecosystem leveraging investments in laser beam processing to enable in-space joining via laser beam welding. A number of ground and flight experiments are being performed to develop laser beam welding as a mature process for use in space. These experiments access varied combinations of reduced gravity, reduced atmospheric pressure, and extreme temperatures to simulate relevant space environments. The associated instrumentation needed to exquisitely understand fundamental mechanisms during laser beam welding and to provide adequate validation datasets for computational models is also being developed and/or integrated.

thermal vacuum↗
NASA NTRSDate not supplied

Establishing an In-Space Joining Ecosystem at NASA Marshall via Laser Beam Welding

NASA Marshall is establishing an ISAM technology development ecosystem leveraging investments in laser beam processing to enable in-space joining via laser beam welding. A number of ground and flight experiments are being performed to develop laser beam welding as a mature process for use in space. These experiments access varied combinations of reduced gravity, reduced atmospheric pressure, and extreme temperatures to simulate relevant space environments. The associated instrumentation needed to exquisitely understand fundamental mechanisms during laser beam welding and to provide adequate validation datasets for computational models is also being developed and/or integrated.

thermal vacuum↗
NASA NTRSDate not supplied

Laser Beam Welding Benchmark Experiments Performed in Reduced Gravity and Vacuum

Laser beam welding (LBW) is affected by the extreme temperatures, reduced pressure, and reduced gravity present in space environments. Gravity and pressure especially influence its melt pool and solidification dynamics. A compact, modular vacuum chamber adaptable to flight platforms from parabolic to orbital currently hosts an experiment to investigate the combined influence of reduced gravity and pressure on LBW. A swappable cartridge contains a rotating platen on which customizable workpieces can be welded under vacuum, greatly increasing experimental throughput. Instrumentation includes weld and thermal cameras observing the process, thermocouples placed on workpieces, accelerometers, and vacuum sensors. Experimental data gathered during the welding process will be combined with post-flight nondestructive evaluation, metallography, and mechanical testing to provide validation datasets for computational modeling. Phase I of this effort involves a parabolic flight campaign in low gravity while an anticipated Phase II would proceed to in-space demonstration to access extended duration microgravity.

in-space welding↗
DOE OSTI2022

Development of Training Materials for Pathologists to Provide Machine Learning Validation Data of Tumor-Infiltrating Lymphocytes in Breast Cancer

The High Throughput Truthing project aims to develop a dataset for validating artificial intelligence and machine learning models (AI/ML) fit for regulatory purposes. The context of this AI/ML validation dataset is the reporting of stromal tumor-infiltrating lymphocytes (sTILs) density evaluations in hematoxylin and eosin-stained invasive breast cancer biopsy specimens. After completing the pilot study, we found notable variability in the sTILs estimates as well as inconsistencies and gaps in the provided training to pathologists. Using the pilot study data and an expert panel, we created custom training materials to improve pathologist annotation quality for the pivotal study. We categorized regions of interest (ROIs) based on their mean sTILs density and selected ROIs with the highest and lowest sTILs variability. In a series of eight one-hour sessions, the expert panel reviewed each ROI and provided verbal density estimates and comments on features that confounded the sTILs evaluation. We aggregated and shaped the comments to identify pitfalls and instructions to improve our training materials. From these selected ROIs, we created a training set and proficiency test set to improve pathologist training with the goal to improve data collection for the pivotal study. We are not exploring AI/ML performance in this paper. Instead, we are creating materials that will train crowd-sourced pathologists to be the reference standard in a pivotal study to create an AI/ML model validation dataset. The issues discussed here are also important for clinicians to understand about the evaluation of sTILs in clinical practice and can provide insight to developers of AI/ML models.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2023

The executive disruption model of tinnitus distress: Model validation in two independent datasets using factor score regression

This study presents the executive disruption model (EDM) of tinnitus distress and subsequently validates it statistically using two independent datasets (the Construction Dataset: n = 96 and the Validation Dataset: n = 200). The conceptual EDM was first operationalised as a structural causal model (construction phase). Then multiple regression was used to examine the effect of executive functioning on tinnitus-related distress (validation phase), adjusting for the additional contributions of hearing threshold and psychological distress. For both datasets, executive functioning negatively predicted tinnitus distress score by a similar amount (the Construction Dataset: β = −3.50, p = 0.13 and the Validation Dataset: β = −3.71, p = 0.02). Theoretical implications and applications of the EDM are subsequently discussed; these include the predictive nature of executive functioning in the development of distressing tinnitus, and the clinical utility of the EDM.

Clarke, Nathan A.↗
DOE OSTI2021

Highly Accurate Experimental Heave Decay Tests with a Floating Sphere: A Public Benchmark Dataset for Model Validation of Fluid–Structure Interaction

Highly accurate and precise heave decay tests on a sphere with a diameter of 300 mm were completed in a meticulously designed test setup in the wave basin in the Ocean and Coastal Engineering Laboratory at Aalborg University, Denmark. The tests were dedicated to providing a rigorous benchmark dataset for numerical model validation. The sphere was ballasted to half submergence, thereby floating with the waterline at the equator when at rest in calm water. Heave decay tests were conducted, wherein the sphere was held stationary and dropped from three drop heights: a small drop height, which can be considered a linear case, a moderately nonlinear case, and a highly nonlinear case with a drop height from a position where the whole sphere was initially above the water. The precision of the heave decay time series was calculated from random and systematic standard uncertainties. At a 95% confidence level, uncertainties were found to be very low—on average only about 0.3% of the respective drop heights. Physical parameters of the test setup and associated uncertainties were quantified. A test case was formulated that closely represents the physical tests, enabling the reader to do his/her own numerical tests. The paper includes a comparison of the physical test results to the results from several independent numerical models based on linear potential flow, fully nonlinear potential flow, and the Reynolds-averaged Navier–Stokes (RANS) equations. A high correlation between physical and numerical test results is shown. The physical test results are very suitable for numerical model validation and are public as a benchmark dataset.

42 ENGINEERING↗
NASA NTRS1992

High resolution infrared datasets useful for validating stratospheric models

An important objective of the High Speed Research Program (HSRP) is to support research in the atmospheric sciences that will improve the basic understanding of the circulation and chemistry of the stratosphere and lead to an interim assessment of the impact of a projected fleet of High Speed Civil Transports (HSCT's) on the stratosphere. As part of this work, critical comparisons between models and existing high quality measurements are planned. These comparisons will be used to test the reliability of current atmospheric chemistry models. Two suitable sets of high resolution infrared measurements are discussed.

Rinsland, Curtis P.↗
DOE OSTI2026

Maximizing efficiency of dataset compression for machine learning potentials with information theory

Machine learning interatomic potentials (MLIPs) balance high accuracy and lower costs compared to density functional theory calculations, but their performance often depends on the size and diversity of training datasets. Large datasets improve model accuracy and generalization but are computationally expensive to produce and train on, while smaller datasets risk discarding rare but important atomic environments and compromising MLIP accuracy/reliability. Here, we develop an information-theoretical framework to quantify the efficiency of dataset compression methods and propose an algorithm that maximizes this efficiency. By framing atomistic dataset compression as an instance of the minimum set cover (MSC) problem over atom-centered environments, our method identifies the smallest subset of structures that contains as much information as possible from the original dataset while pruning redundant information. The approach is extensively demonstrated on the GAP-20 and TM23 datasets and validated on 64 varied datasets from the ColabFit repository. Across all cases, MSC consistently retains outliers, preserves dataset diversity, and reproduces the long-tail distributions of forces even at high compression rates, outperforming other subsampling methods. Furthermore, MLIPs trained on MSC-compressed datasets exhibit reduced error for out-of-distribution data even in low-data regimes. We explain these results using an outlier analysis and show that such quantitative conclusions could not be achieved with conventional dimensionality reduction methods. The algorithm is implemented in the open-source QUESTS package and can be used for several tasks in atomistic modeling, from data subsampling, outlier detection, and training improved MLIPs at a lower cost.

36 MATERIALS SCIENCE↗
DOE OSTI2021

dsgrid (Demand-Side Grid Toolkit) [SWR-21-52]

The demand-side grid model harnesses decades of sector-specific energy modeling expertise to understand current and future U.S. electricity load for power systems analyses. The API provides tools for submitting datasets to dsgrid projects, validating individual datasets and project data as a whole, querying, analysis, and visualization.

Hale, Elaine↗
NASA NTRSDate not supplied

Supervised Machine Learning Approach for Classifying Earth Science Publications

The data collections archived and distributed by the GES DISC NASA data center are widely utilized for various Earth Science studies. As these collections are created, many research works are published regarding these collections' algorithms, their validation, and their applications. As NASA data centers collect these publications for public use, it is helpful to categorize them based on how they relate to their associated datasets. Specifically, whether the publication linked to the GES DISC dataset is using it for applicational research, describing the algorithm used for the dataset creation, validating the dataset, or providing a general overview of the data collection. Currently, this process requires simple manual labeling, and as such, it may be possible to solve via automation. To approach this problem, machine learning classifiers were developed to predict a publication's category. Manually labeled publications were used as the training data for the supervised machine learning algorithms, specifically Random Forest and Multinomial Naïve Bayes. After balancing the dataset and implementing the Multinomial Naïve Bayes algorithm, the classification accuracy achieved was substantially higher than the baseline accuracy, thus significantly improving the efficiency of publication labeling.

Rohan Dayal↗
DOE OSTI2024

Using pile-up collisions as an abundant source of low-energy hadronic physics processes in ATLAS and an extraction of the jet energy resolution

During the 2015–2018 data-taking period, the Large Hadron Collider delivered proton-proton bunch crossings at a centre-of-mass energy of 13 TeV to the ATLAS experiment at a rate of roughly 30 MHz, where each bunch crossing contained an average of 34 independent inelastic proton-proton collisions. The ATLAS trigger system selected roughly 1 kHz of these bunch crossings to be recorded to disk. Offline algorithms then identify one of the recorded collisions as the collision of interest for subsequent data analysis, and the remaining collisions are referred to as pile-up. Pile-up collisions represent a trigger-unbiased dataset, which is evaluated to have an integrated luminosity of 1.33 pb -1 in 2015–2018. This is small compared with the normal trigger-based ATLAS dataset, but when combined with vertex-by-vertex jet reconstruction it provides up to 50 times more dijet events than the conventional single-jet-trigger-based approach, and does so without adding any additional cost or requirements on the trigger system, readout, or storage. The pile-up dataset is validated through comparisons with a special trigger-unbiased dataset recorded by ATLAS, and its utility is demonstrated by means of a measurement of the jet energy resolution in dijet events, where the statistical uncertainty is significantly reduced for jet transverse momenta below 65 GeV.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRS2011

An Overview of the GEOS-5 Aerosol Reanalysis

GEOS-5 is the latest version of the NASA Global Modeling and Assimilation Office (GMAO) earth system model. GEOS-5 contains components for atmospheric circulation and composition (including data assimilation), ocean circulation and biogeochemistry, and land surface processes. In addition to traditional meteorological parameters, GEOS-5 includes modules representing the atmospheric composition, most notably aerosols and tropospheric/stratospheric chemical constituents, taking explicit account of the impact of these constituents on the radiative processes of the atmosphere. MERRA is a NASA meteorological reanalysis for the satellite era (1979-present) using GEOS-5. This project focuses on historical analyses of the hydrological cycle on a broad range of weather and climate time scales. As a first step towards an integrated Earth System Analysis (IESA), the GMAO is extending MERRA with reanalyses for other components of the earth system: land, ocean, bio-geochemistry and atmospheric constituents. In this talk we will present results from the MERRA-driven aerosol reanalysis covering the Aqua period (2003-present). The assimilation of Aerosol Optical Depth (AOD) in GEOS-5 involves very careful cloud screening and homogenization of the observing system by means of a Neural Net scheme that translates MODIS radiances into AERONET calibrated AOD. These measurements are further quality controlled using an adaptive buddy check scheme, and assimilated using the Local Displacement Ensemble (LDE) methodology. For this reanalysis, GEOS-5 runs at a nominal 50km horizontal resolution with 72 vertical layers (top at approx. 8Skm). GEOS-5 is driven by daily biomass burning emissions derived from MODIS fire radiative power retrievals. We will present a summary of our efforts to validate such dataset. The GEOS-5 assimilated aerosol fields are first validated by comparison to independent in-situ measurements (AERONET and PM2.5 surface concentrations). In order to asses aerosol absorption on a global scale, we perform a detailed radiative transfer calculation to simulate the UV aerosol index, comparing our results to OMI measurements. By simulating aerosol attenuated backscatter, we use CALIPSO measurements to evaluate the vertical structure of our aerosol estimates, in particular in regions where we have larger discrepancies with OMI. Finally, the consistency of our AOD estimates with estimates from MISR, MODIS/Deep Blue, OMI and PARASOL will be briefly discussed.

da Silva, Arlindo↗