Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 865 records · Page 48

WETCHIMP-WSL: Intercomparison of wetland methane emissions models over West Siberia

Wetlands are the world's largest natural source of methane, a powerful greenhouse gas. The strong sensitivity of methane emissions to environmental factors such as soil temperature and moisture has led to concerns about potential positive feedbacks to climate change. This risk is particularly relevant at high latitudes, which have experienced pronounced warming and where thawing permafrost could potentially liberate large amounts of labile carbon over the next 100 years. However, global models disagree as to the magnitude and spatial distribution of emissions, due to uncertainties in wetland area and emissions per unit area and a scarcity of in situ observations. Recent intensive field campaigns across the West Siberian Lowland (WSL) make this an ideal region over which to assess the performance of large-scale process-based wetland models in a high-latitude environment. Here we present the results of a follow-up to the Wetland and Wetland CH 4 Intercomparison of Models Project (WETCHIMP), focused on the West Siberian Lowland (WETCHIMP-WSL). We assessed 21 models and 5 inversions over this domain in terms of total CH 4 emissions, simulated wetland areas, and CH 4 fluxes per unit wetland area and compared these results to an intensive in situ CH 4 flux data set, several wetland maps, and two satellite surface water products. We found that (a) despite the large scatter of individual estimates, 12-year mean estimates of annual total emissions over the WSL from forward models (5.34 ± 0.54 Tg CH 4 yr⁻¹), inversions (6.06 ± 1.22 Tg CH 4 yr⁻¹), and in situ observations (3.91 ± 1.29 Tg CH 4 yr⁻¹) largely agreed; (b) forward models using surface water products alone to estimate wetland areas suffered from severe biases in CH 4 emissions; (c) the interannual time series of models that lacked either soil thermal physics appropriate to the high latitudes or realistic emissions from unsaturated peatlands tended to be dominated by a single environmental driver (inundation or air temperature), unlike those of inversions and more sophisticated forward models; (d) differences in biogeochemical schemes across models had relatively smaller influence over performance; and (e) multiyear or multidecade observational records are crucial for evaluating models' responses to long-term climate change.

54 ENVIRONMENTAL SCIENCES↗

An Initial Study of the Sensitivity of Aircraft Vortex Spacing System (AVOSS) Spacing Sensitivity to Weather and Configuration Input Parameters

A study has been performed on a computer code modeling an aircraft wake vortex spacing system during final approach. This code represents an initial engineering model of a system to calculate reduced approach separation criteria needed to increase airport productivity. This report evaluates model sensitivity toward various weather conditions (crosswind, crosswind variance, turbulent kinetic energy, and thermal gradient), code configurations (approach corridor option, and wake demise definition), and post-processing techniques (rounding of provided spacing values, and controller time variance).

Riddick, Stephen E.↗

Assessment of a Hybrid Retrofit Gas Water Heater

This project completed a modeling evaluation of a hybrid gas water heater that combines a reduced capacity tankless unit with a downsized storage tank. This product would meet a significant market need by providing a higher efficiency gas water heater solution for retrofit applications while maintaining compatibility with the 1/2 inch gas lines and standard B vents found in most homes. The TRNSYS simulation tool was used to model a base case 0.60 EF atmospheric gas storage water, a 0.82 EF non-condensing gas tankless water heater, an existing (high capacity) hybrid unit on the market, and an alternative hybrid unit with lower storage volume and reduced gas input requirements. Simulations were completed under a 'peak day' sizing scenario with 183 gpd hot water loads in a Minnesota winter climate case. Full-year simulations were then completed in three climates (ranging from Phoenix to Minneapolis) for three hot water load scenarios (36, 57, and 96 gpd). Model projections indicate that the alternative hybrid offers an average 4.5% efficiency improvement relative to the 0.60 EF gas storage unit across all scenarios modeled. The alternative hybrid water heater evaluated does show promise, but the current low cost of natural gas across much of the country and the relatively small incremental efficiency improvement poses challenges in initially building a market demand for the product.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Assessment of a Hybrid Retrofit Gas Water Heater

This project completed a modeling evaluation of a hybrid gas water heater that combines a reduced capacity tankless unit with a downsized storage tank. This product would meet a significant market need by providing a higher efficiency gas water heater solution for retrofit applications while maintaining compatibility with the 1/2 inch gas lines and standard B vents found in most homes. The TRNSYS simulation tool was used to model a base case 0.60 EF atmospheric gas storage water, a 0.82 EF non-condensing gas tankless water heater, an existing (high capacity) hybrid unit on the market, and an alternative hybrid unit with lower storage volume and reduced gas input requirements. Simulations were completed under a 'peak day' sizing scenario with 183 gpd hot water loads in a Minnesota winter climate case. Full-year simulations were then completed in three climates (ranging from Phoenix to Minneapolis) for three hot water load scenarios (36, 57, and 96 gpd). Model projections indicate that the alternative hybrid offers an average 4.5% efficiency improvement relative to the 0.60 EF gas storage unit across all scenarios modeled. The alternative hybrid water heater evaluated does show promise, but the current low cost of natural gas across much of the country and the relatively small incremental efficiency improvement poses challenges in initially building a market demand for the product.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Model Gap Quantification and Evaluation

In the real world, model gaps always exist because models cannot perfectly match the objective physical plants. Model gaps reflect the integrity of models, which is vital for model validation and calibration. This letter proposes a novel approach for model gap quantification and evaluation. First, a comprehensive metric is developed to quantify dynamic model gaps. Next, the model gaps are evaluated from a panoramic view of the probability distribution of the comprehensive metric for multiple scenarios. Finally, we demonstrate the proposed quantification and evaluation approach through synchronous machine models.

Wang, Shaobu↗

ARM Cloud Radar Simulator Package for Global Climate Models Value-Added Product

It has been challenging to directly compare U.S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Climate Research Facility ground-based cloud radar measurements with climate model output because of limitations or features of the observing processes and the spatial gap between model and the single-point measurements. To facilitate the use of ARM radar data in numerical models, an ARM cloud radar simulator was developed to converts model data into pseudo-ARM cloud radar observations that mimic the instrument view of a narrow atmospheric column (as compared to a large global climate model [GCM] grid-cell), thus allowing meaningful comparison between model output and ARM cloud observations. The ARM cloud radar simulator value-added product (VAP) was developed based on the CloudSat simulator contained in the community satellite simulator package, the Cloud Feedback Model Intercomparison Project (CFMIP) Observation Simulator Package (COSP) (Bodas-Salcedo et al., 2011), which has been widely used in climate model evaluation with satellite data (Klein et al., 2013, Zhang et al., 2010). The essential part of the CloudSat simulator is the QuickBeam radar simulator that is used to produce CloudSat-like radar reflectivity, but is capable of simulating reflectivity for other radars (Marchand et al., 2009; Haynes et al., 2007). Adapting QuickBeam to the ARM cloud radar simulator within COSP required two primary changes: one was to set the frequency to 35 GHz for the ARM Ka-band cloud radar, as opposed to 94 GHz used for the CloudSat W-band radar, and the second was to invert the view from the ground to space so as to attenuate the beam correctly. In addition, the ARM cloud radar simulator uses a finer vertical resolution (100 m compared to 500 m for CloudSat) to resolve the more detailed structure of clouds captured by the ARM radars. The ARM simulator has been developed following the COSP workflow (Figure 1) and using the capabilities available in COSP wherever possible. The ARM simulator is written in Fortran 90, just as is the COSP. It is incorporated into COSP to facilitate use by the climate modeling community. In order to evaluate simulator output, the observational counterpart of the simulator output, radar reflectivity-height histograms (CFAD) is also generated from the ARM observations. This report includes an overview of the ARM cloud radar simulator VAP and the required simulator-oriented ARM radar data product (radarCFAD) for validating simulator output, as well as a user guide for operating the ARM radar simulator VAP.

54 ENVIRONMENTAL SCIENCES↗

Building America Case Study: Assessment of a Hybrid Retrofit Gas Water Heater

This project completed a modeling evaluation of a hybrid gas water heater that combines a reduced capacity tankless unit with a downsized storage tank. This product would meet a significant market need by providing a higher efficiency gas water heater solution for retrofit applications while maintaining compatibility with the half-inch gas lines and standard B vents found in most homes. The TRNSYS simulation tool was used to model a base case 0.60 EF atmospheric gas storage water, a 0.82 EF non-condensing gas tankless water heater, an existing (high capacity) hybrid unit on the market, and an alternative hybrid unit with lower storage volume and reduced gas input requirements. Simulations were completed under a 'peak day' sizing scenario with 183 gpd hot water loads in a Minnesota winter climate case. Full-year simulations were then completed in three climates (ranging from Phoenix to Minneapolis) for three hot water load scenarios (36, 57, and 96 gpd). Model projections indicate that the alternative hybrid offers an average 4.5% efficiency improvement relative to the 0.60 EF gas storage unit across all scenarios modeled. The alternative hybrid water heater evaluated does show promise, but the current low cost of natural gas across much of the country and the relatively small incremental efficiency improvement poses challenges in initially building a market demand for the product.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Integrating Analytical Solutions and U-Net Model for Predicting Groundwater Contaminant Plumes in Pump-and-Treat Systems

Pump-and-treat (P&T) is a common technique for groundwater remediation involving the extraction and treatment of contaminated water above ground. Optimizing the design and operation of the P&T well network is essential for maximizing the system’s effectiveness and efficiency. However, this optimization often necessitates many model evaluations, leading to computationally demanding tasks. This study introduces a novel approach that integrates analytical solutions for groundwater dynamics with the U-Net (Ronneberger et al., 2015) deep learning framework to predict groundwater contaminant plume migration under dynamic pumping conditions. By incorporating the Thiem equation (Thiem, 1906) into the input preprocessing, the U-Net model transforms sparse well data into a continuous spatial field that captures the hydraulic impacts of pumping activities. This integration enables the model to leverage both deep learning capabilities and classical physics-based groundwater theories, enhancing prediction accuracy and computational efficiency. These advancements can facilitate rapid, large-scale evaluations of P&T optimization simulations, allowing for timely and effective decision-making in well placement and system management. We demonstrate the model's robust performance across both simplified transient 2D models and a more complex 3D heterogeneous site model at the 200 West P&T facility at the Hanford Site. The U-Net-based model offers substantial computational advantages, reducing simulation times significantly compared to full physics-based models and providing a powerful tool for rapid site evaluation and P&T system optimization, such as evaluating alternative P&T well network designs. Our findings highlight the potential of advanced machine learning models to significantly enhance the efficiency and sustainability of groundwater remediation efforts, offering a novel application of U-Net architecture in environmental science.

Pump-and-treat↗

A phenomenological model on the kink mode threshold varying with the inclination of sheath boundary

In nature and many laboratory plasmas, a magnetic flux tube threaded by current or a flux rope has a footpoint at a boundary. The current driven kink mode is one of the fundamental ideal magnetohydrodynamic instabilities in plasmas. It has an instability threshold that has been found to strongly depend on boundary conditions (BCs). We provide a theoretical model to explain the transition of this threshold dependence between nonline tied and line tied boundary conditions. We evaluate model parameters using experimentally measured plasma data, explicitly verify several kink eigenfunctions, and validate the model predictions for boundary conditions BCs that span the range between NLT and LT BCs. Based on this model, one could estimate the kink threshold given knowledge of the displacement of a flux rope end, or conversely estimate flux rope end motion based on knowledge of it kink stability threshold.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Earth System Model Improvement Pipeline via Uncertainty Attribution and Active Learning

Primary focal area: 2 (Predictive Modeling via AI): We develop methods to formally quantify uncertainties in Earth System models for the land-atmosphere coupled system. Science Challenge: Earth system models still have significant biases in historical predictions of the intensity and frequency of water cycling extremes (e.g., droughts and flood events), leading to low confidence in future projections. Uncertainties arise from incomplete understanding of land and atmospheric processes, and insufficient observational constraints on model parameters. Many observations, including those from key DOE investments such as ARM and AmeriFlux, are used to evaluate model performance but have not been used to formally quantify model uncertainty because of the expense of running ESM simulations. An efficient pipeline engaging cutting-edge machine learning (ML) and uncertainty quantification (UQ) methods is needed to improve the predictive understanding of water cycle extremes in the Earth system.

54 ENVIRONMENTAL SCIENCES↗

Tree hydrodynamic modelling of the soil–plant–atmosphere continuum using FETCH3

Modelling the water transport along the soil–plant–atmosphere continuum is fundamental to estimating and predicting transpiration fluxes. A Finite-difference Ecosystem-scale Tree Crown Hydrodynamics model (FETCH3) for the water fluxes across the soil–plant–atmosphere continuum is presented here. The model combines the water transport pathways into one vertical dimension, and assumes that the water flow through the soil, roots, and above-ground xylem can be approximated as flow in porous media. This results in a system of three partial differential equations, resembling the Richardson–Richards equation, describing the transport of water through the plant system and with additional terms representing sinks and sources for the transfer of water from the soil to the roots and from the leaves to the atmosphere. The numerical scheme, developed in Python 3, was tested against exact analytical solutions for steady state and transient conditions using simplified but realistic model parameterizations. The model was also used to simulate a previously published case study, where observed transpiration rates were available, to evaluate model performance. With the same model setup as the published case study, FETCH3 results were in agreement with observations. Through a rigorous coupling of soil, root xylem, and stem xylem, FETCH3 can account for variable water capacitance, while conserving mass and the continuity of the water potential between these three layers. FETCH3 provides a ready-to-use open access numerical model for the simulation of water fluxes across the soil–plant–atmosphere continuum.

54 ENVIRONMENTAL SCIENCES↗

Trapped Radiation Model Uncertainties: Model-Data and Model-Model Comparisons

The standard AP8 and AE8 models for predicting trapped proton and electron environments have been compared with several sets of flight data to evaluate model uncertainties. Model comparisons are made with flux and dose measurements made on various U.S. low-Earth orbit satellites (APEX, CRRES, DMSP. LDEF, NOAA) and Space Shuttle flights, on Russian satellites (Photon-8, Cosmos-1887, Cosmos-2044), and on the Russian Mir space station. This report gives the details of the model-data comparisons -- summary results in terms of empirical model uncertainty factors that can be applied for spacecraft design applications are given in a companion report. The results of model-model comparisons are also presented from standard AP8 and AE8 model predictions compared with the European Space Agency versions of AP8 and AE8 and with Russian trapped radiation models.

Armstrong, T. W.↗

Trapped Radiation Model Uncertainties: Model-Data and Model-Model Comparisons

The standard AP8 and AE8 models for predicting trapped proton and electron environments have been compared with several sets of flight data to evaluate model uncertainties. Model comparisons are made with flux and dose measurements made on various U.S. low-Earth orbit satellites (APEX, CRRES, DMSP, LDEF, NOAA) and Space Shuttle flights, on Russian satellites (Photon-8, Cosmos-1887, Cosmos-2044), and on the Russian Mir Space Station. This report gives the details of the model-data comparisons-summary results in terms of empirical model uncertainty factors that can be applied for spacecraft design applications are given in a combination report. The results of model-model comparisons are also presented from standard AP8 and AE8 model predictions compared with the European Space Agency versions of AP8 and AE8 and with Russian-trapped radiation models.

Armstrong, T. W.↗

Improving thermodynamic nudging in the E3SM Atmosphere Model version 2 (EAMv2): strategy and hindcast skills on weather systems

Nudging techniques are commonly employed to constrain atmospheric simulations toward observed states, facilitating model evaluation and sensitivity studies. However, if applied improperly – particularly to thermodynamic variables such as temperature and humidity – nudging can distort physical processes and introduce spurious biases, undermining the credibility of the simulations. This study presents an improved nudging implementation that applies vertically modulated tendencies to reduce adverse impacts on model physics. The framework is tested in version 2 of the Energy Exascale Earth System Model (EAMv2) using a suite of hindcast simulations nudged toward ERA5 reanalysis. We systematically evaluate the individual and combined effects of nudging wind, temperature, and humidity fields on the model's ability to represent large-scale atmospheric states and high-impact weather systems. Results show that the revised strategy – particularly when nudging temperature and humidity at selected levels – enhances hindcast skill by improving agreement with ERA5 without degrading the hydrological cycle or precipitation processes. Additional improvements in surface temperature, outgoing longwave radiation, and precipitation biases are achieved through targeted nudging of land-surface variables. The proposed approach strengthens the representation of large-scale conditions relevant to tropical cyclones, atmospheric rivers, and extratropical cyclones in the low-resolution EAMv2. These findings demonstrate that carefully designed thermodynamic nudging, especially of temperature and humidity, improves the realism of constrained simulations and broadens the utility of nudged EAMv2 for atmospheric modeling, machine learning, and high-impact weather research.

Atmospheric river↗

Artificial Intelligence (AI) Methods for Augmenting the IMPACT Tool Evidence Library

Development of the Evidence Library for use with the IMPACT probability risk assessment tool took several years and involved a staggering amount of effort from a multi-disciplinary team. A very significant amount of the labor effort to collect, assess and finalize the Clinical Finding Form (CliFF) for each of the 119 medical conditions was provided by physician subject matter experts from the Exploration Medical Capability (ExMC) Element Clinical and Science Team. Many AI tools such as ChatGPT are excellent at summarizing large amounts of information and the current project was initiated to determine how such tools might streamline laborious processes, e.g., review and summarization of many scientific research publications, to execute key steps more efficiently in the process of developing CliFFs. The process for collecting the evidence which is found in the CliFFs is well documented in the Evidence Library Methods document (ELM; HRP-48036*). Using ELM and the CliFF development instructions as a guideline, a team of developers is leveraging Microsoft Azure AI tools and services along with open-source frameworks, to construct an AI-assisted automated pipeline. This pipeline is designed to search, retrieve, and process the necessary data sources, and ultimately help generate the final version of a CliFF. Currently, the large language model evaluates the relevance of each source material to spaceflights, either as direct evidence or as an analog. Additionally, the model assists in extracting keywords and generating brief summaries to enhance augmented retrieval and search processes in later stages of CliFF development. Once the data is ready, the model can perform semantic search and retrieval, generating and extracting valuable information for the CliFF. For instance, it can handle epidemiological statistical data, such as incidence rates and the likelihood of best or worst-case scenarios. The steps that required reading and summarizing articles were viewed as providing the greatest return on investment since large language models are very efficient and accurate in summarizing large amounts of text. Since labor effort to complete the original CliFF was not recorded with sufficient granularity, comparisons with an AI tool-generated CliFF will provide merely an approximation of time saved. Upon completion of the process, the CliFF for the medical condition “appendicitis” generated with the support of AI-based methods will serve as a proof-of-concept and will be compared to the original appendicitis CliFF to determine if use of the tools resulted in content and conclusory similarity. Based upon the results from face validation of the two CliFFs, modifications to the process will be made if necessary and additional condition CliFFs will be evaluated. Ultimately, CliFFs for the entire set of medical conditions will be created with the assistance of AI tools. Depending on the cost savings realized, CliFFs for additional medical conditions can be created to expand the Evidence Library. Future direction includes specifying the characteristics of the reviewer (prompting the AI tools to generate output assuming the reviewer is a sub-specialist physician, or nurse or EMT/medic) to determine if the effects on AI-generated output are different based on knowledge, skills and abilities. *Exploration Medical Capability Evidence Library Methods, HRP-48036 Rev A, July 2022.

Ali Al↗

Evaluating human performance modeling for system assessment: Promise and problems

The development and evaluation of computational human performance models is examined. An intention is to develop models which can be used to interact with system prototypes and simulations to perform system assessment. Currently LR is working on a set of models emulating cognitive, psychomotor, auditory, and visual activity for multiple operator positions of a command and control simulation system. These models, developed in conjunction with BBN Systems and Technologies, function within the simulation environment and allow for both unmanned system assessment and manned (human-in-loop) assessment of system interface and team interactions. These are relatively generic models with built-in flexibility which allows modification of some model parameters. These models have great potential for improving the efficiency and effectiveness of system design, test, and evaluation. However, the extent of the practical utility of these models is unclear. Initial verification efforts comparing model performance within the simulation to actual human operators on a similar, independent simulation have been performed and current efforts are directed at comparing human and model performance within the same simulation environment.

Patterson, Robert W.↗

A comparative study of two-way and offline coupled WRF v3.4 and CMAQ v5.0.2 over the contiguous US: performance evaluation and impacts of chemistry–meteorology feedbacks on air quality

The two-way coupled Weather Research and Forecasting and Community Multiscale Air Quality (WRF-CMAQ) model has been developed to more realistically represent the atmosphere by accounting for complex chemistry–meteorology feedbacks. In this study, we present a comparative analysis of two-way (with consideration of both aerosol direct and indirect effects) and offline coupled WRF v3.4 and CMAQ v5.0.2 over the contiguous US. Long-term (5 years from 2008 to 2012) simulations using WRF-CMAQ with both offline and two-way coupling modes are carried out with anthropogenic emissions based on multiple years of the U.S. National Emission Inventory and chemical initial and boundary conditions derived from an advanced Earth system model (i.e., a modified version of the Community Earth System Model/Community Atmospheric Model). The comprehensive model evaluations show that both two-way WRF-CMAQ and WRF-only simulations perform well for major meteorological variables such as temperature at 2 m, relative humidity at 2 m, wind speed at 10 m, precipitation (except for against the National Climatic Data Center data), and shortwave and longwave radiation. Both two-way and offline CMAQ also show good performance for ozone (O3) and fine particulate matter (PM 2.5 ). Due to the consideration of aerosol direct and indirect effects, two-way WRF-CMAQ shows improved performance over offline coupled WRF and CMAQ in terms of spatiotemporal distributions and statistics, especially for radiation, cloud forcing, O 3 , sulfate, nitrate, ammonium, elemental carbon, tropospheric O 3 residual, and column nitrogen dioxide (NO 2 ). For example, the mean biases have been reduced by more than 10 W m –2 for shortwave radiation and cloud radiative forcing and by more than 2 ppb for max 8 h O 3 . However, relatively large biases still exist for cloud predictions, some PM 2.5 species, and PM 10 that warrant follow-up studies to better understand those issues. The impacts of chemistry–meteorological feedbacks are found to play important roles in affecting regional air quality in the US by reducing domain-average concentrations of carbon monoxide (CO), O 3 , nitrogen oxide (NO x ), volatile organic compounds (VOCs), and PM 2.5 by 3.1 % (up to 27.8 %), 4.2 % (up to 16.2 %), 6.6 % (up to 50.9 %), 5.8 % (up to 46.6 %), and 8.6 % (up to 49.1 %), respectively, mainly due to reduced radiation, temperature, and wind speed. The overall performance of the two-way coupled WRF-CMAQ model achieved in this work is generally good or satisfactory and the improved performance for two-way coupled WRF-CMAQ should be considered along with other factors in developing future model applications to inform policy making.

58 GEOSCIENCES↗