Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,351 records · Page 75

Climate Model Diagnostic and Evaluation: With a Focus on Satellite Observations

Each year, we host a summer school that brings together the next generation of climate scientists - about 30 graduate students and postdocs from around the world - to engage with premier climate scientists from the Jet Propulsion Laboratory and elsewhere. Our yearly summer school focuses on topics on the leading edge of climate science research. Our inaugural summer school, held in 2011, was on the topic of "Using Satellite Observations to Advance Climate Models," and enabled students to explore how satellite observations can be used to evaluate and improve climate models. Speakers included climate experts from both NASA and the National Oceanic and Atmospheric Administration (NOAA), who provided updates on climate model diagnostics and evaluation and remote sensing of the planet. Details of the next summer school will be posted here in due course.

climate↗

Evaluation of CMIP5 Model Precipitation Using PERSIANN-CDR

Abstract The purpose of this study is to use the PERSIANN–Climate Data Record (PERSIANN-CDR) dataset to evaluate the ability of 32 CMIP5 models in capturing the behavior of daily extreme precipitation estimates globally. The daily long-term historical global PERSIANN-CDR allows for a global investigation of eight precipitation indices that is unattainable with other datasets. Quantitative comparisons against CPC daily gauge; GPCP One-Degree Daily (GPCP1DD); and TRMM 3B42, version 7 (3B42V7), datasets show the credibility of PERSIANN-CDR to be used as the reference data for global evaluation of CMIP5 models. This work uniquely defines different study regions by partitioning global land areas into 25 groups based on continent and climate zone type. Results show that model performance in warm temperate and equatorial regions in capturing daily extreme precipitation behavior is largely mixed in terms of index RMSE and correlation, suggesting that these regions may benefit from weighted model averaging schemes or model selection as opposed to simple model averaging. The three driest climate regions (snow, polar, and arid) exhibit high correlations and low RMSE values when compared against PERSIANN-CDR estimates, with the exceptions of the cold regions showing an inability to capture the 95th and 99th percentile annual total precipitation characteristics. A comprehensive assessment of each model’s performance in each continent–climate zone defined group is provided as a guide for both model developers to target regions and processes that are not yet fully captured in certain climate types, and for climate model output users to be able to select the models and/or the study areas that may best fit their applications of interest.

Nguyen, Phu↗

Evaluation of CMIP6 Model Simulations of PM 2.5 and its Components Over China

Earth system models (ESMs) participating in the latest Coupled Model Intercomparison Project Phase 6 (CMIP6) simulate various components of fine particulate matter (PM 2.5 ) as major climate forcers. Yet the model performance for PM 2.5 components remains little evaluated due in part to a lack of observational data. Here, we evaluate near-surface concentrations of PM 2.5 and its five main components over China as simulated by 14 CMIP6 models, including organic carbon (OC; available in 14 models), black carbon (BC; 14 models), sulfate (14 models), nitrate (4 models), and ammonium (5 models). For this purpose, we collect observational data between 2000 and 2014 from a satellite-based dataset for total PM2.5 and from 2469 measurement records in the literature for PM 2.5 components. Seven models output total PM 2.5 concentrations, and they all underestimate the observed total PM 2.5 over eastern China, with GFDL-ESM4 (−1.5 %) and MPI-ESM-1-2-HAM (−1.1 %) exhibiting the smallest biases averaged over the whole country. The other seven models, for which we recalculate total PM 2.5 from the available component output, underestimate the total PM 2.5 concentrations partly because of the missing model representations of nitrate and ammonium. Concentrations of the five individual components are underestimated in almost all models, except that sulfate is overestimated in MPI-ESM-1-2-HAM by 12.6 % and in MRI-ESM2-0 by 24.5 %. The underestimation is the largest for OC (by −71.2 % to −37.8 % across the 14 models) and the smallest for BC (−47.9 % to −12.1 %). The multi-model mean (MMM) reproduces the observed spatial pattern for OC (R = 0.51), sulfate (R = 0.57), nitrate (R = 0.70) and ammonium (R = 0.74) fairly well, yet the agreement is poorer for BC (R = 0.39). The varying performances of ESMs on total PM 2.5 and its components have important implications for the modeled magnitude and spatial pattern of aerosol radiative forcing.

CMIP6↗

Evaluation of Regression Models of Balance Calibration Data Using an Empirical Criterion

An empirical criterion for assessing the significance of individual terms of regression models of wind tunnel strain gage balance outputs is evaluated. The criterion is based on the percent contribution of a regression model term. It considers a term to be significant if its percent contribution exceeds the empirical threshold of 0.05%. The criterion has the advantage that it can easily be computed using the regression coefficients of the gage outputs and the load capacities of the balance. First, a definition of the empirical criterion is provided. Then, it is compared with an alternate statistical criterion that is widely used in regression analysis. Finally, calibration data sets from a variety of balances are used to illustrate the connection between the empirical and the statistical criterion. A review of these results indicated that the empirical criterion seems to be suitable for a crude assessment of the significance of a regression model term as the boundary between a significant and an insignificant term cannot be defined very well. Therefore, regression model term reduction should only be performed by using the more universally applicable statistical criterion.

Ulbrich, Norbert↗

ESMValTool (v1.0) – a community diagnostic and performance metrics tool for routine evaluation of Earth system models in CMIP

A community diagnostics and performance metrics tool for the evaluation of Earth system models (ESMs) has been developed that allows for routine comparison of single or multiple models, either against predecessor versions or against observations. The priority of the effort so far has been to target specific scientific themes focusing on selected essential climate variables (ECVs), a range of known systematic biases common to ESMs, such as coupled tropical climate variability, monsoons, Southern Ocean processes, continental dry biases, and soil hydrology–climate interactions, as well as atmospheric CO 2 budgets, tropospheric and stratospheric ozone, and tropospheric aerosols. The tool is being developed in such a way that additional analyses can easily be added. A set of standard namelists for each scientific topic reproduces specific sets of diagnostics or performance metrics that have demonstrated their importance in ESM evaluation in the peer-reviewed literature. The Earth System Model Evaluation Tool (ESMValTool) is a community effort open to both users and developers encouraging open exchange of diagnostic source code and evaluation results from the Coupled Model Intercomparison Project (CMIP) ensemble. This will facilitate and improve ESM evaluation beyond the state-of-the-art and aims at supporting such activities within CMIP and at individual modelling centres. Ultimately, we envisage running the ESMValTool alongside the Earth System Grid Federation (ESGF) as part of a more routine evaluation of CMIP model simulations while utilizing observations available in standard formats (obs4MIPs) or provided by the user.

58 GEOSCIENCES↗

Model Data Archive for Manuscript Titled "Evaluation of a Coupled Surface–Subsurface Hydrologic Model Using Dense Water‑Level Sensors in a Mixed Urban–Rural Watershed"

This archive provides scripts, input files, and datasets used for the implementation and evaluation of a fully coupled surface–subsurface hydrologic model in the Neches River Basin, southeast Texas. The study uses the Advanced Terrestrial Simulator (ATS) to simulate coupled surface–subsurface hydrologic processes over a mixed urban–rural watershed and evaluates model performance using a dense network of 136 in situ water-level sensors, nine U.S. Geological Survey (USGS) stream gauges, and SSEBop-derived evapotranspiration estimates during the period October 2014–June 2024. The workflow is implemented primarily in Python 3 using the Watershed Workflow package. The Jupyter notebooks can be executed using open-source software such as Anaconda JupyterLab or Visual Studio Code. Other data files include TXT, CSV, XML, SHP, TIF, NetCDF, HDF5, and ExodusII files, which can be processed using the provided Python scripts. ATS input files are provided in XML format and can be edited using any commonly used text editor. This archive contains: *Scripts and input files used to generate the ATS model setup, including watershed discretization, mesh generation, parameter mapping, and model configuration. *Jupyter notebooks used for preprocessing observational data, evaluating streamflow, water levels, and evapotranspiration, computing performance metrics, and generating the figures presented in the manuscript. *ATS simulation outputs and processed observational datasets, including OneRain and DD6 water-level sensors, USGS streamflow observations, GIS data, and supporting spatial datasets used throughout the study.

Dense water-level sensor network↗

Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3

Systematic, routine, and comprehensive evaluation of Earth system models (ESMs) facilitates benchmarking improvement across model generations and identifying the strengths and weaknesses of different model configurations. By gauging the consistency between models and observations, this endeavor is becoming increasingly necessary to objectively synthesize the thousands of simulations contributed to the Coupled Model Intercomparison Project (CMIP) to date. The Program for Climate Model Diagnosis and Intercomparison (PCMDI) Metrics Package (PMP) is an open-source Python software package that provides quick-look objective comparisons of ESMs with one another and with observations. The comparisons include metrics of large- to global-scale climatologies, tropical inter-annual and intra-seasonal variability modes such as the El Niño–Southern Oscillation (ENSO) and Madden–Julian Oscillation (MJO), extratropical modes of variability, regional monsoons, cloud radiative feedbacks, and high-frequency characteristics of simulated precipitation, including its extremes. The PMP comparison results are produced using all model simulations contributed to CMIP6 and earlier CMIP phases. An important objective of the PMP is to document the performance of ESMs participating in the recent phases of CMIP, together with providing version-controlled information for all datasets, software packages, and analysis codes being used in the evaluation process. Among other purposes, this also enables modeling groups to assess performance changes during the ESM development cycle in the context of the error distribution of the multi-model ensemble. Quantitative model evaluation provided by the PMP can assist modelers in their development priorities. In this paper, we provide an overview of the PMP, including its latest capabilities, and discuss its future direction.

54 ENVIRONMENTAL SCIENCES↗

Evaluation of integrated assessment model hindcast experiments: a case study of the GCAM 3.0 land use module

Abstract. Hindcasting experiments (conducting a model forecast for a time period in which observational data are available) are being undertaken increasingly often by the integrated assessment model (IAM) community, across many scales of models. When they are undertaken, the results are often evaluated using global aggregates or otherwise highly aggregated skill scores that mask deficiencies. We select a set of deviation-based measures that can be applied on different spatial scales (regional versus global) to make evaluating the large number of variable–region combinations in IAMs more tractable. We also identify performance benchmarks for these measures, based on the statistics of the observational dataset, that allow a model to be evaluated in absolute terms rather than relative to the performance of other models at similar tasks. An ideal evaluation method for hindcast experiments in IAMs would feature both absolute measures for evaluation of a single experiment for a single model and relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models, such as in community intercomparison studies. The performance benchmarks highlight the use of this scheme for model evaluation in absolute terms, providing information about the reasons a model may perform poorly on a given measure and therefore identifying opportunities for improvement. To demonstrate the use of and types of results possible with the evaluation method, the measures are applied to the results of a past hindcast experiment focusing on land allocation in the Global Change Assessment Model (GCAM) version 3.0. The question of how to more holistically evaluate models as complex as IAMs is an area for future research. We find quantitative evidence that global aggregates alone are not sufficient for evaluating IAMs that require global supply to equal global demand at each time period, such as GCAM. The results of this work indicate it is unlikely that a single evaluation measure for all variables in an IAM exists, and therefore sector-by-sector evaluation may be necessary.

58 GEOSCIENCES↗

Toward real-time optimization through model reduction and model discrepancy sensitivities

Optimization problems arise in a range of scenarios, from optimal control to model parameter estimation. In many applications, such as the development of digital twins, it is essential to solve these optimization problems within wall-clock-time limitations. However, this is often unattainable for complex systems, such as those modeled by nonlinear partial differential equations. One strategy for mitigating this issue is to construct a reduced-order model (ROM) that enables more rapid optimization. In particular, the use of nonintrusive ROMs—those that do not require access to the full-order model at evaluation time—is popular because they facilitate the computation of optimization solutions within the wall-clock time requirements. However, the optimization solution will be unreliable if the iterates move outside the ROM training data. This article proposes the use of hyper-differential sensitivity analysis with respect to model discrepancy (HDSA-MD) as a computationally efficient tool to augment ROM-constrained optimization and improve its reliability. The proposed approach consists of two phases: (i) an offline phase where several full-order model evaluations are computed to train the ROM, and (ii) an online phase where a ROM-constrained optimization problem is solved, a limited number of full-order model evaluations are computed, and HDSA-MD is used to enhance the optimization solution. Numerical results are demonstrated for two examples, atmospheric contaminant control and wildfire ignition location estimation, in which a ROM is trained offline using inaccurate atmospheric data. In conclusion, the HDSA-MD update yields a significant improvement in the ROM-constrained optimization solution using only one full-order model evaluation online with corrected atmospheric data.

PDE-constrained optimization↗

Impacts of benchmarking choices on inferred model skill of the Arctic–Boreal terrestrial carbon cycle

Abstract Land surface models require continuous validation against observations to improve and reduce simulation uncertainty. However, inferred model performance can be heavily influenced by subjective choices made in the selection and application of observational data products. A key area often misrepresented by models is the Arctic–Boreal region, which is a potential tipping point region in Earth’s climate system due to large permafrost carbon stocks that are vulnerable to release with climate warming. We use the International Land Model Benchmarking (ILAMB) framework to evaluate how the model skill of TRENDY-v9 models varies based on the choice of observational-based benchmark and how benchmarks are applied in model evaluation. This analysis uses global datasets integrated into ILAMB and new, regionally-specific observational products from the Arctic–Boreal Vulnerability Experiment. Our results cover the overall time period of 1979–2019 and show that model scores can vary substantially depending on the data product applied, with higher model scores indicating better model performance against observations. The lowest model scores occur when benchmarked against regional, compared to global, datasets. We also evaluate observed and modeled functional relationships between ecosystem respiration and air temperature and between gross primary production and precipitation. Here, we find that the magnitude and shape of the responses are strongly impacted by the choice of observational dataset and the approach used to construct the functional relationship benchmark. These results suggest that model evaluation studies could conclude a false sense of model skill if only using a single benchmark data product or if not applying regional data products when performing a regional model analysis. Collectively, our findings highlight the influence of benchmarking choices on model evaluation and point to the need for benchmarking guidelines when assessing model skill.

Poe, Jeralyn (ORCID:0000000318495278)↗

“Full Model” Nuclear Data and Covariance Evaluation Process Using TALYS, Total Monte Carlo and Backward-forward Monte Carlo

The “Full model” evaluation process, that is used in CEA DAM DIF to evaluate nuclear data in the continuum region, makes extended use of nuclear models implemented in the TALYS code to account for experimental data (both differential and integral) by varying the parameters of these models until a satisfactory description of these experimental data is reached. For the evaluation of the covariance data associated with this evaluated data, the Backward-forward Monte Carlo (BFMC) method was devised in such a way that it mirrors the process of the “Full model” evaluation method. When coupled with the Total Monte Carlo method via the T6 system developed by NRG Petten, the BFMC method allows to make use of integral experiments to constrain the distribution of model parameters, and hence the distribution of derived observables and their covariance matrix. Together, TALYS, TMC, BFMC, and T6, constitute a powerful integrated tool for nuclear data evaluation, that allows for evaluation of nuclear data and the associated covariance matrix, all at once, making good use of all the available experimental information to drive the distribution of the model parameters and the derived observables.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Rule based design of conceptual models for formative evaluation

A Human-Computer Interface (HCI) Prototyping Environment with embedded evaluation capability has been investigated. This environment will be valuable in developing and refining HCI standards and evaluating program/project interface development, especially Space Station Freedom on-board displays for payload operations. This environment, which allows for rapid prototyping and evaluation of graphical interfaces, includes the following four components: (1) a HCI development tool, (2) a low fidelity simulator development tool, (3) a dynamic, interactive interface between the HCI and the simulator, and (4) an embedded evaluator that evaluates the adequacy of a HCI based on a user's performance. The embedded evaluation tool collects data while the user is interacting with the system and evaluates the adequacy of an interface based on a user's performance. This paper describes the design of conceptual models for the embedded evaluation system using a rule-based approach.

Moore, Loretta A.↗

Rule based design of conceptual models for formative evaluation

A Human-Computer Interface (HCI) Prototyping Environment with embedded evaluation capability has been investigated. This environment will be valuable in developing and refining HCI standards and evaluating program/project interface development, especially Space Station Freedom on-board displays for payload operations. This environment, which allows for rapid prototyping and evaluation of graphical interfaces, includes the following four components: (1) a HCI development tool; (2) a low fidelity simulator development tool; (3) a dynamic, interactive interface between the HCI and the simulator; and (4) an embedded evaluator that evaluates the adequacy of a HCI based on a user's performance. The embedded evaluation tool collects data while the user is interacting with the system and evaluates the adequacy of an interface based on a user's performance. This paper describes the design of conceptual models for the embedded evaluation system using a rule-based approach.

Moore, Loretta A.↗

Evaluating Effectiveness of Modeling Motion System Feedback in the Enhanced Hess Structural Model of the Human Operator

In order to use the Hess Structural Model to predict the need for certain cueing systems, George and Cardullo significantly expanded it by adding motion feedback to the model and incorporating models of the motion system dynamics, motion cueing algorithm and a vestibular system. This paper proposes a methodology to evaluate effectiveness of these innovations by performing a comparison analysis of the model performance with and without the expanded motion feedback. The proposed methodology is composed of two stages. The first stage involves fine-tuning parameters of the original Hess structural model in order to match the actual control behavior recorded during the experiments at NASA Visual Motion Simulator (VMS) facility. The parameter tuning procedure utilizes a new automated parameter identification technique, which was developed at the Man-Machine Systems Lab at SUNY Binghamton. In the second stage of the proposed methodology, an expanded motion feedback is added to the structural model. The resulting performance of the model is then compared to that of the original one. As proposed by Hess, metrics to evaluate the performance of the models include comparison against the crossover models standards imposed on the crossover frequency and phase margin of the overall man-machine system. Preliminary results indicate the advantage of having the model of the motion system and motion cueing incorporated into the model of the human operator. It is also demonstrated that the crossover frequency and the phase margin of the expanded model are well within the limits imposed by the crossover model.

Zaychik, Kirill↗

On the Realism of the Rain Microphysics Representation of a Squall Line in the WRF Model. Part I: Evaluation with Multifrequency Cloud Radar Doppler Spectra Observations

This study investigates how multifrequency cloud radar observations can be used to evaluate the representation of rain microphysics in the WRF Model using two bulk microphysics schemes. A squall line observed over Oklahoma on 12 June 2011 is used as a case study. A recently developed retrieval technique combining observations of two vertically pointing cloud radars provides quantitative description of the drop size distribution (DSD) properties of the transition and stratiform regions of the squall-line system. For the first time, the results of this multifrequency cloud radar retrieval are compared to more conventional retrievals from a nearby polarimetric radar, and a supplementary result of this work is that this new methodology provides a much more detailed description of the DSD vertical and temporal variations. While the extent and evolution of the squall line is well reproduced by the model, the 1-h low-reflectivity transition region is not. In the stratiform region, simulations with both schemes are able to reproduce the observed downdraft and the associated significative subsaturation below the melting level, but with a slight overestimation of the relative humidity. Under this subsaturated air, the simulated rain mixing ratio continuously decreases toward the ground, in agreement with the observations. Conversely, the profiles of the mean volume diameter and the concentration parameter of the DSDs are not well reproduced. These discrepancies pinpoint at an issue in the representation of rain microphysics. The companion paper, investigates the sources of the biases in the microphysics processes in the rain layer by performing numerical sensitivity studies.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Advanced Models for Nondestructive Evaluation of Aging Nuclear Power Plant Cables

Background: Nuclear power plant (NPP) cables must function correctly for proper power supply and control of the nuclear reactor. This is especially important for public and environmental safety at critical times such as during reactor shut-down following loss of coolant. Over extended periods of service, the cable insulation and jacket polymer materials suffer from degradation due to exposure to environmental stressors such as heat and ionizing radiation. The focus of this project has been the development of detailed descriptions of the way in which insulation polymers degrade, from a materials science perspective, and the development of new electrical techniques to monitor cable polymer condition nondestructively. Objectives: The objectives of this project have been to i) perform detailed characterization experiments upon cable insulation polymers that have experienced thermal and radiation exposure mimicking that received during NPP operation, including the exploration of novel cable nondestructive evaluation methods; capacitive, THz and infrared methods; ii) identify the most sensitive indicators of aging in these materials, and iii) develop advanced, validated models describing microstructural and chemical changes that are observed in cable insulation polymers due to thermal and radiation exposure. Methods: Accelerated aging was performed on the two most common United States NPP cable insulation polymers (cross-linked polyethylene (XLPE) and ethylene propylene rubber (EPR)) under various conditions of elevated temperature and gamma radiation dose. Aging mechanisms were elucidated through extensive thermal, mechanical and microscopic characterization of the aged samples, and materials modeling. Dielectric and IR spectra were measured on pristine and aged samples. Data was analyzed statistically, via multiple linear regression, partial least squares and probabilistic neural network approaches. Impact: Extant cable monitoring techniques do not deal successfully with all polymer types. This research has laid a foundation upon which new electrical nondestructive test methods for cable polymer condition monitoring may be developed, hence launching the development of new test methods for cable insulation monitoring based on measuring electrical rather than mechanical parameters. Collaboration: This project has benefitted from collaboration between experts in materials science and engineering, materials modeling, electrical engineering, and nondestructive evaluation. The majority of the characterization measurements were carried out at Iowa State University. Materials modeling was carried out primarily at Washington State University. Irradiation of samples took place at Pacific Northwest National Laboratory (PNNL), where a student intern from Iowa State was hosted for one summer, setting up the aging experiments. A research student from the University of Bologna, Italy, joined the project for their thesis research at Iowa State University with samples provided by PNNL. The project benefitted from regular involvement of senior engineers in US-based industry partners. Project investigators presented findings at various national and international conferences and workshops.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Microelectronics reliability : physics-of-failure based modeling and lifetime evaluation

This handbook presents a physics-of-failure approach to microelectronics reliability modeling and assessment. Knowledge of the root cause and physical behavior of key failure mechanisms in microelectronic devices has improved dramatically over recent years and has led to the development of more sophisticated reliability modeling tools and techniques...

White, Mark↗