Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

The Single Column Atmosphere Model Version 6 (SCAM6): Not a Scam but a Tool for Model Evaluation and Development

The Single Column Atmosphere Model (SCAM) is a single column model version of the Community Atmosphere Model (CAM). Here we describe the functionality and features of SCAM6, available as part of CAM6 in the Community Earth System Model, version 2 (CESM2). SCAM6 features a wide selection of standard cases, as well as the ability to easily configure a case specified by the user based on a particular point in a CAM 3-D simulation. This work illustrates how SCAM6 reproduces CAM6 results for physical parameterizations, mostly of moisture and clouds. We demonstrate how SCAM6 can be used for model development through different physics selections, as well as with parameter sweep experiments to highlight the sensitivity of cloud properties to the specification of the vapor deposition process in the cloud microphysics. Furthermore, we use SCAM6 to illustrate the sensitivity of CAM6 cloud radiative properties and precipitation to variable drop number (cloud microphysics properties). Finally, we illustrate how SCAM6 can be used to explore critical emergent processes such as cloud feedbacks and show that CAM6 cloud responses to surface warming in stratus and stratocumulus regimes are similar to those in CAM5. CAM6 has a larger response in the shallow cumulus regime than CAM5. CAM6 cloud feedbacks in the shallow cumulus regime are sensitive to turbulence parameters. SCAM6 is thus a valuable tool for model development, evaluation, and scientific analy sis and an important part of the model hierarchy in Community Earth System Model, version 2.

58 GEOSCIENCES↗

Model evaluation, recommendation and prioritizing of future work for the manipulator emulator testbed

The Manipulator Emulator Testbed (MET) is to provide a facility capable of hosting the simulation of various manipulator configurations to support concept studies, evaluation, and other engineering development activities. Specifically, the testbed is intended to support development of the Space Station Remote Manipulator System (SSRMS) and related systems. The objective of this study is to evaluate the math models developed for the MET simulation of a manipulator's rigid body dynamics and the servo systems for each of the driven manipulator joints. Specifically, the math models are examined with regard to their amenability to pipeline and parallel processing. Based on this evaluation and the project objectives, a set of prioritized recommendations are offered for future work.

Kelly, Frederick A.↗

Climate Information and Lessons Learned from the NASA Earth eXchange (NEX), Short-term Prediction Research and Transition Center (NASA SPoRT), and the NASA Regional Climate Model Evaluation System (RCMES)

Climate resilience can emerge from a combination of (1) climate projections that enable future-oriented planning and (2) contemporary observations that can be used for adaptation in the face of realized climate. NASA is uniquely positioned to provide global climate projections and satellite-based observations to users who must take a global perspective on climate risk (e.g., supply chain managers). Here, we will share the capabilities and experiences of three NASA projects. The first project, the NASA Earth eXchange (NEX) provides two downscaled climate projections, NEX-GDDP-CMIP6, a global 25 km resolution product, and NEX-DCP30-CMIP6, an 800 m resolution product covering the coterminous U.S. specifically designed for U.S. National Climate Assessment (NCA). NEX also offers land surface products that mirror the twice-daily products provided by the Moderate Resolution Imaging Spectroradiometer (MODIS) but are derived from operational geostationary weather satellite data collected at 5-10 minutes intervals. The second project, the Short-term Prediction Research and Transition Center (SPoRT), was established in 2002 to transition unique NASA satellite products and capabilities to stakeholders to improve decision-making that benefits society. SPoRT provides products and capabilities derived from NASA observations and models to end users in government, academia, and the private sector. SPoRT translates NASA Earth science into actionable solutions reaching over 400 stakeholders across 80 organizations. Finally, the Regional Climate Model Evaluation System (RCMES) is designed to facilitate regional-scale evaluations of climate models by providing standardized access to a vast and comprehensive set of observations, such as satellite, in-situ, and reanalyses, and modeling resources such as CMIP & CORDEX. These three projects form part of the core capabilities at the NASA Centers deliver climate information under the new NASA Earth Science to Action (ES2A) strategy.

NEX↗

PCMDI - An Earth System Model Evaluation Project Lawrence Livermore National Laboratory Annual Report

This annual report describes the activities of PCMDI – An Earth System Model Evaluation Project for the period November 1, 2019 – June 30, 2020. PCMDI is a project at Lawrence Livermore National Laboratory that is funded by the Climate and Environmental Sciences Division (CESD) of the Department of Energy. PCMDI is funded by two efforts within CESD: The Regional and Global Model Analysis program area (Dr. Renu Joseph, program manager) and the Atmospheric Systems Research program (Dr. Shaima Nasiri and Jeffrey Stehr, program managers). As instructed by Drs. Joseph, Nasiri, and Stehr, we provide an annual report addressing sections 4 (Progress and Plans) and 5 (Staffing and Budget Summary) of the SFA Program Management and Performance Reporting Criteria.

54 ENVIRONMENTAL SCIENCES↗

On the Forward Modeling of Radar Doppler Spectrum Width From LES: Implications for Model Evaluation

Large-eddy simulations of an observed single-layer Arctic mixed-phase cloud are analyzed to study the value of forward modeling of profiling millimeter wave cloud radar Doppler spectral width for model evaluation. Individual broadening terms and their uncertainties are quantified for the observed spectral width and compared to modeled broadening terms. Modeled turbulent broadening is narrower than the observed values when the turbulent kinetic energy dissipation rate from the subgrid scale model is used in the forward model. The total dissipation rates, estimated with the subgrid scale dissipation rates and the numerical dissipation rates, agree much better with both the retrieved dissipation rates and those inferred from the power spectra of the simulated vertical air velocity. The comparison of the microphysical broadening provides another evaluative measure of the ice properties in the simulation. To accurately retrieve dissipation rates as well as each broadening term from the observations, we suggest a few modifications to previously presented techniques. First, we show that the inertial subrange spectrum filtered with the radar sampling volume is a better underlying model than the unfiltered -5/3 law for the retrieval of the dissipation rate from the power spectra of the mean Doppler velocity. Second, we demonstrate that it is important to filter out turbulence and remove the layer-mean reflectivity-weighted mean fall speed from the observed mean Doppler velocity to avoid overestimation of shear broadening. Finally, we provide a method to quantify the uncertainty in the retrieved dissipation rates, which eventually propagates to the uncertainty in the microphysical broadening.

54 ENVIRONMENTAL SCIENCES↗

Data assimilation and model evaluation experiment datasets

The Institute for Naval Oceanography, in cooperation with Naval Research Laboratories and universities, executed the Data Assimilation and Model Evaluation Experiment (DAMEE) for the Gulf Stream region during fiscal years 1991-1993. Enormous effort has gone into the preparation of several high-quality and consistent datasets for model initialization and verification. This paper describes the preparation process, the temporal and spatial scopes, the contents, the structure, etc., of these datasets. The goal of DAMEE and the need of data for the four phases of experiment are briefly stated. The preparation of DAMEE datasets consisted of a series of processes: (1) collection of observational data; (2) analysis and interpretation; (3) interpolation using the Optimum Thermal Interpolation System package; (4) quality control and re-analysis; and (5) data archiving and software documentation. The data products from these processes included a time series of 3D fields of temperature and salinity, 2D fields of surface dynamic height and mixed-layer depth, analysis of the Gulf Stream and rings system, and bathythermograph profiles. To date, these are the most detailed and high-quality data for mesoscale ocean modeling, data assimilation, and forecasting research. Feedback from ocean modeling groups who tested this data was incorporated into its refinement. Suggestions for DAMEE data usages include (1) ocean modeling and data assimilation studies, (2) diagnosis and theoretical studies, and (3) comparisons with locally detailed observations.

Lai, Chung-Cheng A.↗

ARM data-oriented metrics and diagnostics package for climate model evaluation (ARM-DIAGS-V3) version 3

A Python-based metrics and diagnostics package is currently being developed by the ARM Infrastructure Team at Lawrence Livermore National Laboratory to facilitate the use of long-term high frequency measurements from the ARM program in evaluating the regional climate simulation of clouds, radiation, precipitation, and aerosols. This metrics and diagnostics package computes climatological means of targeted climate model simulation and generates tables and plots for comparing the model simulation with ARM observational data. The CMIP model data sets are also included in the package to enable model inter-comparison as demonstrated in Zhang et al. (2018) and Zhang et al. (2020). The mean of the CMIP model can be served as a reference for individual models. Basic performance metrics are computed to measure the accuracy of mean state and variability of climate models. The evaluated physical quantities include cloud fraction, temperature, relative humidity, cloud liquid water path, total column water vapor, precipitation, sensible and latent heat fluxes, aerosol optical depth, and radiative fluxes, with plan to extend to more fields, such as the evaluation of model simulated aerosol physicochemical properties and cloud microphysics properties. Process-oriented diagnostics focusing on aerosol, cloud, and precipitation-related phenomena are also being developed for the evaluation and development of specific model physical parameterizations. In addition to the Southern Great Plains (SGP), North Slope of Alaska (NSA) and Tropical Western Pacific (TWP) atmospheric observatories in the ARMDIAGS version 2.0, the version 3.0 package have extended to the data collected at the ARM Eastern North Atlantic (ENA) atmospheric observatory and the Observation and Modeling of the Green Ocean Amazon (GOAMAZON) field campaign. The metrics and diagnostics package are currently built upon standard Python libraries and additional Python packages developed by DOE (CDAT). The ARM metrics and diagnostic package is available publicly with the hope that it can serve as an easy entry point for climate modelers to compare their models with ARM data. In this report, we first provide an overview of major metrics in section 2. The input data, which constitutes the core content of the metrics and diagnostics package, is summarized in section 3. A user's guide documenting the workflow/structure of the version 3.0 codes and including step-by-step instruction for running the package is described in section 4.

54 ENVIRONMENTAL SCIENCES↗

Model Evaluation Report for LNG Dispersion on Fire Dynamics Simulator Version 6.5.3

This report is an assessment of the computational fluid dynamics (CFD) code Fire Dynamics Simulator (FDS), version 6.5.3, using the Model Evaluation Protocol (MEP) for Liquefied Natural Gas (LNG). The MEP consists of model validation and a scientific assessment that encompasses model information and model verification activities. Model validation entails comparison against various field and wind-tunnel trials for LNG and other dense gases with and without obstructions. The results indicate that FDS is generally over-predictive for most of the field scale dense gas releases and cases involving obstructions without requiring a safety factor of 2, however, since there are uncertainties in extrapolating to accident scales a safety factor of 2 (1/2 LFL criteria) may be appropriate. The results of this scientific assessment and prior verification efforts demonstrate the soundness of the numerical techniques and robustness of FDS. The validation results indicate that FDS is suitable for modeling dense gas dispersion with and without obstructions.

03 NATURAL GAS↗

A Model Evaluation Data Set for the Tropical ARM Sites

This data set has been derived from various ARM and external data sources with the main aim of providing modelers easy access to quality controlled data for model evaluation. The data set contains highly aggregated (in time) data from a number of sources at the tropical ARM sites at Manus and Nauru. It spans the years of 1999 and 2000. The data set contains information on downward surface radiation; surface meteorology, including precipitation; atmospheric water vapor and cloud liquid water content; hydrometeor cover as a function of height; and cloud cover, cloud optical thickness and cloud top pressure information provided by the International Satellite Cloud Climatology Project (ISCCP).

54 ENVIRONMENTAL SCIENCES↗

Aerosols at the Poles: An Aerocom Phase II Multi-Model Evaluation

Atmospheric aerosols from anthropogenic and natural sources reach the polar regions through long-range transport and affect the local radiation balance. Such transport is, however, poorly constrained in present-day global climate models, and few multi-model evaluations of polar anthropogenic aerosol radiative forcing exist. Here we compare the aerosol optical depth (AOD) at 550 nm from simulations with 16 global aerosol models from the AeroCom Phase II model intercomparison project with available observations at both poles. We show that the annual mean multi-model median is representative of the observations in Arctic, but that the intermodel spread is large. We also document the geographical distribution and seasonal cycle of the AOD for the individual aerosol species: black carbon (BC) from fossil fuel and biomass burning, sulfate, organic aerosols (OAs), dust, and sea-salt. For a subset of models that represent nitrate and secondary organic aerosols (SOAs), we document the role of these aerosols at high latitudes. The seasonal dependence of natural and anthropogenic aerosols differs with natural aerosols peaking in winter (seasalt) and spring (dust), whereas AOD from anthropogenic aerosols peaks in late spring and summer. The models produce a median annual mean AOD of 0.07 in the Arctic (defined here as north of 60 degrees N). The models also predict a noteworthy aerosol transport to the Antarctic (south of 70 degrees S) with a resulting AOD varying between 0.01 and 0.02. The models have estimated the shortwave anthropogenic radiative forcing contributions to the direct aerosol effect (DAE) associated with BC and OA from fossil fuel and biofuel (FF), sulfate, SOAs, nitrate, and biomass burning from BC and OA emissions combined. The Arctic modelled annual mean DAE is slightly negative (-0.12 W m(exp. -2), dominated by a positive BC FF DAE in spring and a negative sulfate DAE in summer. The Antarctic DAE is governed by BC FF. We perform sensitivity experiments with one of the AeroCom models (GISS modelE) to investigate how regional emissions of BC and sulfate and the lifetime of BC influence the Arctic and Antarctic AOD. A doubling of emissions in eastern Asia results in a 33 percent increase in Arctic AOD of BC. A doubling of the BC lifetime results in a 39 percent increase in Arctic AOD of BC. However, these radical changes still fall within the AeroCom model range.

Sand, Maria↗

Aerosols at the poles: an AeroCom Phase II multi-model evaluation

Abstract. Atmospheric aerosols from anthropogenic and natural sources reach the polar regions through long-range transport and affect the local radiation balance. Such transport is, however, poorly constrained in present-day global climate models, and few multi-model evaluations of polar anthropogenic aerosol radiative forcing exist. Here we compare the aerosol optical depth (AOD) at 550 nm from simulations with 16 global aerosol models from the AeroCom Phase II model intercomparison project with available observations at both poles. We show that the annual mean multi-model median is representative of the observations in Arctic, but that the intermodel spread is large. We also document the geographical distribution and seasonal cycle of the AOD for the individual aerosol species: black carbon (BC) from fossil fuel and biomass burning, sulfate, organic aerosols (OAs), dust, and sea-salt. For a subset of models that represent nitrate and secondary organic aerosols (SOAs), we document the role of these aerosols at high latitudes.The seasonal dependence of natural and anthropogenic aerosols differs with natural aerosols peaking in winter (sea-salt) and spring (dust), whereas AOD from anthropogenic aerosols peaks in late spring and summer. The models produce a median annual mean AOD of 0.07 in the Arctic (defined here as north of 60° N). The models also predict a noteworthy aerosol transport to the Antarctic (south of 70° S) with a resulting AOD varying between 0.01 and 0.02. The models have estimated the shortwave anthropogenic radiative forcing contributions to the direct aerosol effect (DAE) associated with BC and OA from fossil fuel and biofuel (FF), sulfate, SOAs, nitrate, and biomass burning from BC and OA emissions combined. The Arctic modelled annual mean DAE is slightly negative (−0.12 W m−2), dominated by a positive BC FF DAE in spring and a negative sulfate DAE in summer. The Antarctic DAE is governed by BC FF. We perform sensitivity experiments with one of the AeroCom models (GISS modelE) to investigate how regional emissions of BC and sulfate and the lifetime of BC influence the Arctic and Antarctic AOD. A doubling of emissions in eastern Asia results in a 33 % increase in Arctic AOD of BC. A doubling of the BC lifetime results in a 39 % increase in Arctic AOD of BC. However, these radical changes still fall within the AeroCom model range.

54 ENVIRONMENTAL SCIENCES↗

Taking climate model evaluation to the next level

We report that earth system models are complex and represent a large number of processes, resulting in a persistent spread across climate projections for a given future scenario. Owing to different model performances against observations and the lack of independence among models, there is now evidence that giving equal weight to each available model projection is suboptimal. This Perspective discusses newly developed tools that facilitate a more rapid and comprehensive evaluation of model simulations with observations, process-based emergent constraints that are a promising way to focus evaluation on the observations most relevant to climate projections, and advanced methods for model weighting. Lastly, these approaches are needed to distill the most credible information on regional climate changes, impacts, and risks for stakeholders and policy-makers.

54 ENVIRONMENTAL SCIENCES↗

System integration of wind and solar power in integrated assessment models: A cross-model evaluation of new approaches

Mitigation-Process Integrated Assessment Models (MP-IAMs) are used to analyze long-term transformation pathways of the energy system required to achieve stringent climate change mitigation targets. Due to their substantial temporal and spatial aggregation, IAMs cannot explicitly represent all detailed challenges of integrating the variable renewable energies (VRE) wind and solar in power systems, but rather rely on parameterized modeling approaches. In the ADVANCE project, six international modeling teams have developed new approaches to improve the representation of power sector dynamics and VRE integration in IAMs. In this study, we qualitatively and quantitatively evaluate the last years' modeling progress and study the impact of VRE integration modeling on VRE deployment in IAM scenarios. For a comprehensive and transparent qualitative evaluation, we first develop a framework of 18 features of power sector dynamics and VRE integration. We then apply this framework to the newly-developed modeling approaches to derive a detailed map of strengths and limitations of the different approaches. For the quantitative evaluation, we compare the IAMs to the detailed hourly-resolution power sector model REMIX. We find that the new modeling approaches manage to represent a large number of features of the power sector, and the numerical results are in reasonable agreement with those derived from the detailed power sector model. Updating the power sector representation and the cost and resources of wind and solar substantially increased wind and solar shares across models: Under a carbon price of 30$/tCO2 in 2020 (increasing by 5% per year), the model-average cost-minimizing VRE share over the period 2050-2100 is 62% of electricity generation, 24%-points higher than with the old model version.

(IAM) Variable Renewable Energy (VRE)↗

Ground heat flux: An analytical review of 6 models evaluated at 88 sites and globally

Uncertainty in ground heat flux (G) means that evaluation of the other terms in the surface energy balance (e.g., latent and sensible heat fluxes (LE and H)) remains problematic. Algorithms that calculate LE and H require available energy, the difference between net radiation, R NET , and G. There are a wide range of approaches to model G for large-scale applications, with a subsequent wide range of estimates and accuracies. Here, we provide the largest review of these methods to date (N = 6), evaluating modeled G against measured G from 88 FLUXNET sites. The instantaneous midday variability in G is best captured by models forced with net radiation, while models forced by temperature show the least error at both instantaneous and daily time scales. We produce global decadal data sets of G to illustrate regional and seasonal sensitivities, as well as uncertainty. Global model mean midmorning instantaneous G is highest during September, October, and November at 63.42 (±16.84) Wm -2 , while over December, January, and February G is lowest at 53.86 (±18.09) Wm -2 but shows greater intermodel uncertainty. Results from this work have the potential to improve evapotranspiration estimates and guide appropriate G model selection and development for various land uses.

58 GEOSCIENCES↗

An inverse problem strategy based on forward model evaluations: Gradient-based optimization without adjoint solves

This study presents a new nonlinear programming formulation for the solution of inverse problems. First, a general inverse problem formulation based on the compliance error functional is presented. The proposed error functional enables the computation of the Lagrange multipliers, and thus the first order derivative information, at the expense of just one model evaluation. Therefore, the calculation of the Lagrange multipliers does not require the solution of the computationally intensive adjoint problem. This leads to significant speedups for large-scale, gradient-based inverse problems.

42 ENGINEERING↗

Evaluating modeling tools for the EDOS

The Earth Observing System (EOS) Data and Operations System (EDOS) Project is developing a functional, system performance model to support the system implementation phase of the EDOS which is being designed and built by the Goddard Space Flight Center (GSFC). The EDOS Project will use modeling to meet two key objectives: (1) manage system design impacts introduced by unplanned changed in mission requirements; and (2) evaluate evolutionary technology insertions throughout the development of the EDOS. To select a suitable modeling tool, the EDOS modeling team developed an approach for evaluating modeling tools and languages by deriving evaluation criteria from both the EDOS modeling requirements and the development plan. Essential and optional features for an appropriate modeling tool were identified and compared with known capabilities of several modeling tools. Vendors were also provided the opportunity to model a representative EDOS processing function to demonstrate the applicability of their modeling tool to the EDOS modeling requirements. This paper emphasizes the importance of using a well defined approach for evaluating tools to model complex systems like the EDOS. The results of this evaluation study do not in any way signify the superiority of any one modeling tool since the results will vary with the specific modeling requirements of each project.

Knoble, Gordon↗