Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2

New model framework and structure and the commonality evaluation model

The development of a framework and structure for shuttle era unmanned spacecraft projects and the development of a commonality evaluation model is documented. The methodology developed for model utilization in performing cost trades and comparative evaluations for commonality studies is discussed. The model framework consists of categories of activities associated with the spacecraft system's development process. The model structure describes the physical elements to be treated as separate identifiable entities. Cost estimating relationships for subsystem and program-level components were calculated.

Source record↗

Development of evaluation models of manpower needs for dismantling the dry conversion process-related equipment in uranium refining and conversion plant (URCP)

Evaluation models for determining the manpower needs for dismantling various types of equipment in uranium refining and conversion plant (URCP) have been developed. The models are widely applicable to other uranium handling facilities. Additionally, a simplified model was developed for easily and accurately calculating the manpower needs for dismantling dry conversion process-related equipment (DP equipment). It is important to evaluate beforehand project management data such as manpower needs to prepare an optimized decommissioning plan and implement effective dismantling activity. The Japan Atomic Energy Agency (JAEA) has developed the project management data evaluation system for dismantling activities (PRODIA code), which can generate project management data using evaluation models. For preparing an optimized decommissioning plan, these evaluation models should be established based on the type of nuclear facility and actual dismantling data. In URCP, the dry conversion process of reprocessed uranium and others was operated until 1999, and the equipment related to the main process was dismantled from 2008 to 2011. Actual data such as manpower for dismantling were collected during the dismantling activities, and evaluation models were developed using the collected actual data on the basis of equipment classification considering the characteristics of uranium handling facility. (authors)

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Nevada National Security Site (NNSS) Environmental Remediation Using EPA's Model Evaluation Guidance to Move to Closure - 15588

The National Nuclear Security Administration Nevada Field Office (NNSA/NFO) has adapted the Environmental Protection Agency's guidance on environmental model evaluation as a framework for evaluating corrective action investigation models. Model evaluation is an important confidence building activity, which is needed to support closure of the Nevada National Security Site (NNSS) Corrective Action Units (CAUs) associated with underground nuclear testing. Through this process, site-investigation models of groundwater flow and contaminant transport are reviewed and compared to new site-specific data to determine if the results can be used to support regulatory decisions required for closure and to achieve NNSA/NFO site management objectives. The regulatory decisions include identifying and establishing CAU regulatory boundaries; identifying institutional controls, including use-restriction boundaries; and developing a long-term closure monitoring program. A successful case study from Frenchman Flat, Nevada is used to illustrate the model evaluation process. Groundwater flow and transport from two of the 10 underground nuclear tests in Frenchman Flat had the largest potential for contaminating groundwater, and were selected for evaluation. Wells were drilled, logged, hydraulically tested, sampled, and analyzed for radionuclides and the data was evaluated with respect to previous site conceptualization and numerical model results. The approach and results of this process as applied to the PIN STRIPE test and how it helped build confidence to close the Frenchman Flat CAU is described in this case study. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Model Evaluation and Hindcasting: An Experiment with an Integrated Assessment Model

Integrated assessment models have been extensively used for analyzing long term energy and greenhouse emissions trajectories and have influenced key policies on this subject. Though admittedly these models are focused on the long term trajectories, how well these models are able to capture historical dynamics is an open question. In a first experiment of its kind, we present a framework for evaluation of such integrated assessment models. We use Global Change Assessment Model for this zero order experiment, and focus on the building sector results for USA. We calibrate the model for 1990 and run it forward up to 2095 in five year time steps. This gives us results for 1995, 2000, 2005 and 2010 which we compare to observed historical data at both fuel level and service level. We focus on bringing out the key insights for the wider process of model evaluation through our experiment with GCAM. We begin with highlighting that creation of an evaluation dataset and identification of key evaluation metric is the foremost challenge in the evaluation process. Our analysis highlights that estimation of functional form of the relationship between energy service demand, which is an unobserved variable, and its drivers is a significant challenge in the absence of adequate historical data for both the dependent and driver variables. Historical data availability for key metrics is a serious limiting factor in the process of evaluation. Interestingly, service level data against which such models need to be evaluated are itself a result of models. Thus for energy services, the best we can do is compare our model results with other model results rather than observed and measured data. We show that long term models, by the nature of their construction, will most likely underestimate the rapid growth in some services observed in a short time span. Also, we learn that modeling saturated energy services like space heating is easier than modeling unsaturated services like space cooling and understanding that how far a service is from its saturation level is a key question which we probably don’t have an answer to. Finally and most importantly, even if long term models partially miss the short term dynamics, the long term insights provides by these models is fairly robust. We conclude by highlighting that our work is the first step in the much wider process of integrated assessment model evaluation and will hence have its own limitations. Future evaluation research work should build upon this zero order experiment for improving our modeling of human and coupled earth systems.

building energy↗

Completion Report for Model Evaluation Well ER-11-2: Corrective Action Unit 98: Frenchman Flat

Model Evaluation Well ER-11-2 was drilled for the U.S. Department of Energy, National Nuclear Security Administration Nevada Site Office in support of Nevada Environmental Management Operations at the Nevada National Security Site (formerly known as the Nevada Test Site). The well was drilled in August 2012 as part of a model evaluation program in the Frenchman Flat area of Nye County, Nevada. The primary purpose of the well was to provide detailed geologic, hydrogeologic, chemical, and radionuclide data that can be used to test and build confidence in the applicability of the Frenchman Flat Corrective Action Unit flow and transport models for their intended purpose. In particular, this well was designed to provide data to evaluate the uncertainty in model forecasts of contaminant migration from the upgradient underground nuclear test PIN STRIPE, conducted in borehole U-11b in 1966. Well ER-11-2 will provide information that can be used to refine the Phase II Frenchman Flat hydrostratigraphic framework model if necessary, as well as to support future groundwater flow and transport modeling. The main 31.1-centimeter (cm) hole was drilled to a total depth of 399.6 meters (m). A completion casing string was not set in Well ER-11-2. However, a piezometer string was installed in the 31.1-cm open hole. The piezometer is composed of 7.3-cm stainless-steel tubing hung on 6.0-cm carbon-steel tubing via a crossover sub. The piezometer string was landed at 394.5 m, for monitoring the lower tuff confining unit. Data collected during and shortly after hole construction include composite drill cuttings samples collected every 3.0 m, various geophysical logs, water quality (including tritium and other test-related radionuclides) measurements, and water level measurements. The well penetrated 42.7 m of Quaternary and Tertiary alluvium and 356.9 m of Tertiary volcanic rock. The water-level measured in the piezometer string on September 25, 2012, was 353.8 m below ground surface. No tritium above levels detectable by field methods were encountered in this hole. No well development or hydrologic testing was conducted in this well immediately after completion, and future well development, sampling, and hydrologic testing planned for this well will be limited due to the diameter of the piezometer string. The stratigraphy, general lithology, and the water level are as expected, but the section of geology encountered is higher than expected due to faulting. No tritium above the minimum detection limit of the field equipment was detected because the target aquifer (the Topopah Spring aquifer) at Well ER-11-2 is structurally higher than expected and thus unsaturated.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Completion Report for Model Evaluation Well ER-5-5: Corrective Action Unit 98: Frenchman Flat

Model Evaluation Well ER-5-5 was drilled for the U.S. Department of Energy, National Nuclear Security Administration Nevada Site Office in support of Nevada Environmental Management Operations at the Nevada National Security Site (formerly known as the Nevada Test Site). The well was drilled in July and August 2012 as part of a model evaluation well program in the Frenchman Flat area of Nye County, Nevada. The primary purpose of the well was to provide detailed geologic, hydrogeologic, chemical, and radiological data that can be used to test and build confidence in the applicability of the Frenchman Flat Corrective Action Unit flow and transport models for their intended purpose. In particular, this well was designed to obtain data to evaluate the uncertainty in model forecasts of contaminant migration from the upgradient underground nuclear test MILK SHAKE, conducted in Emplacement Hole U-5k in 1968, which were considered to be uncertain due to the unknown extent of a basalt lava-flow aquifer present in this area. Well ER-5-5 is expected to provide information to refine the Phase II Frenchman Flat hydrostratigraphic framework model, if necessary, as well as to support future groundwater flow and transport modeling. The 31.1-centimeter (cm) diameter hole was drilled to a total depth of 331.3 meters (m). The completion string, set at the depth of 317.2 m, consists of 16.8-cm stainless-steel casing hanging from 19.4-cm carbon-steel casing. The 16.8-cm stainless-steel casing has one slotted interval open to the basalt lava-flow aquifer and limited intervals of the overlying and underlying alluvial aquifer. A piezometer string was also installed in the annulus between the completion string and the borehole wall. The piezometer is composed of 7.3-cm stainless-steel tubing suspended from 6.0-cm carbon-steel tubing. The piezometer string was landed at 319.2 m, to monitor the basalt lava-flow aquifer. Data collected during and shortly after hole construction include composite drill cuttings samples collected every 3.0 m, various geophysical logs, preliminary water quality measurements, and water-level measurements. The well penetrated 331.3 m of Quaternary–Tertiary alluvium, including an intercalated layer of saturated basalt lava rubble. No well development or hydrologic testing was conducted in this well immediately after completion; however, a preliminary water level was measured in the piezometer string at the depth of 283.4 m on September 25, 2012. No tritium above the minimum detection limit of the field instruments was detected in this hole. Future well development, sampling, and hydrologic testing planned for this well will provide more accurate hydrologic information for this site. The stratigraphy, general lithology, and water level were as expected, though the expected basalt lava-flow aquifer is basalt rubble and not the dense, fractured lava as modeled. The lack of tritium transport is likely due to the difference in hydraulic properties of the basalt lava-flow rubble encountered in the well, compared to those of the fractured aquifer used in the flow and transport models.

58 GEOSCIENCES↗

Ten Years of Cloud Properties from MODIS: Global Statistics and Use in Climate Model Evaluation

The NASA Moderate Resolution Imaging Spectroradiometer (MODIS), launched onboard the Terra and Aqua spacecrafts, began Earth observations on February 24, 2000 and June 24,2002, respectively. Among the algorithms developed and applied to this sensor, a suite of cloud products includes cloud masking/detection, cloud-top properties (temperature, pressure), and optical properties (optical thickness, effective particle radius, water path, and thermodynamic phase). All cloud algorithms underwent numerous changes and enhancements between for the latest Collection 5 production version; this process continues with the current Collection 6 development. We will show example MODIS Collection 5 cloud climatologies derived from global spatial . and temporal aggregations provided in the archived gridded Level-3 MODIS atmosphere team product (product names MOD08 and MYD08 for MODIS Terra and Aqua, respectively). Data sets in this Level-3 product include scalar statistics as well as 1- and 2-D histograms of many cloud properties, allowing for higher order information and correlation studies. In addition to these statistics, we will show trends and statistical significance in annual and seasonal means for a variety of the MODIS cloud properties, as well as the time required for detection given assumed trends. To assist in climate model evaluation, we have developed a MODIS cloud simulator with an accompanying netCDF file containing subsetted monthly Level-3 statistical data sets that correspond to the simulator output. Correlations of cloud properties with ENSO offer the potential to evaluate model cloud sensitivity; initial results will be discussed.

Platnick, Steven E.↗

Incorporation of aerosol into the COSPv2 satellite lidar simulator for climate model evaluation

Abstract. Atmospheric aerosol has substantial impacts on climate, air quality and biogeochemical cycles, and its concentrations are highly variable in space and time. A key variability to evaluate within models that simulate aerosol is the vertical distribution, which influences atmospheric heating profiles and aerosol–cloud interactions, to help constrain aerosol residence time and to better represent the magnitude of simulated impacts. To ensure a consistent comparison between modeled and observed vertical distribution of aerosol, we implemented an aerosol lidar simulator within the Cloud Feedback Model Intercomparison Project (CFMIP) Observation Simulator Package version 2 (COSPv2). We assessed the attenuated total backscattered (ATB) signal and the backscatter ratios (SRs) at 532 nm in the U.S. Department of Energy's Energy Exascale Earth System Model version 1 (E3SMv1). The simulator performs the computations at the same vertical resolution as the Cloud-Aerosol Lidar with Orthogonal Polarization (CALIOP), making use of aerosol optics from the E3SMv1 model as inputs and assuming that aerosol is uniformly distributed horizontally within each model grid box. The simulator applies a cloud masking and an aerosol detection threshold to obtain the ATB and SR profiles that would be observed above clouds by CALIOP with its aerosol detection capability. Our analysis shows that the aerosol distribution simulated at a seasonal timescale is generally in good agreement with observations. Over the Southern Ocean, however, the model does not produce the SR maximum as observed in the real world. Comparison between clear-sky and all-sky SRs shows little differences, indicating that the cloud screening by potentially incorrect model clouds does not affect the mean aerosol signal averaged over a season. This indicates that the differences between observed and simulated SR values are due not to sampling errors, but to deficiencies in the representation of aerosol in models. Finally, we highlight the need for future applications of lidar observations at multiple wavelengths to provide insights into aerosol properties and distribution and their representation in Earth system models.

54 ENVIRONMENTAL SCIENCES↗

Global Gridded Crop Model Evaluation: Benchmarking, Skills, Deficiencies and Implications.

Crop models are increasingly used to simulate crop yields at the global scale, but so far there is no general framework on how to assess model performance. Here we evaluate the simulation results of 14 global gridded crop modeling groups that have contributed historic crop yield simulations for maize, wheat, rice and soybean to the Global Gridded Crop Model Intercomparison (GGCMI) of the Agricultural Model Intercomparison and Improvement Project (AgMIP). Simulation results are compared to reference data at global, national and grid cell scales and we evaluate model performance with respect to time series correlation, spatial correlation and mean bias. We find that global gridded crop models (GGCMs) show mixed skill in reproducing time series correlations or spatial patterns at the different spatial scales. Generally, maize, wheat and soybean simulations of many GGCMs are capable of reproducing larger parts of observed temporal variability (time series correlation coefficients (r) of up to 0.888 for maize, 0.673 for wheat and 0.643 for soybean at the global scale) but rice yield variability cannot be well reproduced by most models. Yield variability can be well reproduced for most major producing countries by many GGCMs and for all countries by at least some. A comparison with gridded yield data and a statistical analysis of the effects of weather variability on yield variability shows that the ensemble of GGCMs can explain more of the yield variability than an ensemble of regression models for maize and soybean, but not for wheat and rice. We identify future research needs in global gridded crop modeling and for all individual crop modeling groups. In the absence of a purely observation-based benchmark for model evaluation, we propose that the best performing crop model per crop and region establishes the benchmark for all others, and modelers are encouraged to investigate how crop model performance can be increased. We make our evaluation system accessible to all crop modelers so that other modeling groups can also test their model performance against the reference data and the GGCMI benchmark.

wheat↗

Evaluating Models of Human Performance: Safety-Critical Systems Applications

This presentation is part of panel discussion on Evaluating Models of Human Performance. The purpose of this panel is to discuss the increasing use of models in the world today and specifically focus on how to describe and evaluate models of human performance. My presentation will focus on discussions of generating distributions of performance, and the evaluation of different strategies for humans performing tasks with mixed initiative (Human-Automation) systems. I will also discuss issues with how to provide Human Performance modeling data to support decisions on acceptability and tradeoffs in the design of safety critical systems. I will conclude with challenges for the future.

Feary, Michael S.↗

Incorporating an advanced aerosol activation parameterization into WRF-CAM5: Model evaluation and parameterization intercomparison

Aerosol activation into cloud droplets is an important process that governs aerosol indirect effects. The advanced treatment of aerosol activation by Fountoukis and Nenes (2005) and its recent updates, collectively called the FN series, have been incorporated into a newly developed regional coupled climate-air quality model based on the Weather Research and Forecasting model with the physics package of the Community Atmosphere Model version 5 (WRF-CAM5) to simulate aerosol-cloud interactions in both resolved and convective clouds. The model is applied to East Asia for two full years of 2005 and 2010. A comprehensive model evaluation is performed for model predictions of meteorological, radiative, and cloud variables, chemical concentrations, and column mass abundances against satellite data and surface observations from air quality monitoring sites across East Asia. The model performs overall well for major meteorological variables including near-surface temperature, specific humidity, wind speed, precipitation, cloud fraction, precipitable water, downward shortwave and longwave radiation, and column mass abundances of CO, SO 2 , NO 2 , HCHO, and O 3 in terms of both magnitudes and spatial distributions. Larger biases exist in the predictions of surface concentrations of CO and NO x at all sites and SO 2 , O 3 , PM 2.5 , and PM 10 concentrations at some sites, aerosol optical depth, cloud condensation nuclei over ocean, cloud droplet number concentration (CDNC), cloud liquid and ice water path, and cloud optical thickness. Compared with the default Abdul-Razzack Ghan (2000) parameterization, simulations with the FN series produce ~107–113% higher CDNC, with half of the difference attributable to the higher aerosol activation fraction by the FN series and the remaining half due to feedbacks in subsequent cloud microphysical processes. With the higher CDNC, the FN series are more skillful in simulating cloud water path, cloud optical thickness, downward shortwave radiation, shortwave cloud forcing, and precipitation. Here, the model evaluation identifies several areas of improvements including emissions and their vertical allocation as well as model formulations such as aerosol formation, cloud droplet nucleation, and ice nucleation.

54 ENVIRONMENTAL SCIENCES↗

Data–Driven Velocity Model Evaluation Using K–Means Clustering

In this work, we develop a data-driven clustering method to evaluate a velocity model using surface wave velocity dispersion. This is done by first computing theoretical dispersion curves for 1-D velocity profiles of all the grid locations and then splitting the resulting dispersion curves into a certain number of groups via the K-means clustering. The observed dispersion curves are also clustered following the same procedure and the velocity model is assessed by comparing the spatial patterns obtained for the observed and synthetic data sets. The method is applied to evaluate two community velocity models in southern California, CVM-S4.26 and CVM-H15.1, using phase velocity maps derived for 3–16 s Rayleigh waves. We found a good correlation in the spatial distribution of clusters between the result of CVM-S4.26 and that of the observed data, suggesting that the CVM-S4.26 fits the observed dispersion maps better than the CVM-H15.1 in terms of features extracted from the clustering analysis.

58 GEOSCIENCES↗

Towards improved and more routine Earth system model evaluation in CMIP

The Coupled Model Intercomparison Project (CMIP) has successfully provided the climate community with a rich collection of simulation output from Earth system models (ESMs) that can be used to understand past climate changes and make projections and uncertainty estimates of the future. Confidence in ESMs can be gained because the models are based on physical principles and reproduce many important aspects of observed climate. More research is required to identify the processes that are most responsible for systematic biases and the magnitude and uncertainty of future projections so that more relevant performance tests can be developed. At the same time, there are many aspects of ESM evaluation that are well established and considered an essential part of systematic evaluation but have been implemented ad hoc with little community coordination. Given the diversity and complexity of ESM analysis, we argue that the CMIP community has reached a critical juncture at which many baseline aspects of model evaluation need to be performed much more efficiently and consistently. We provide a perspective and viewpoint on how a more systematic, open, and rapid performance assessment of the large and diverse number of models that will participate in current and future phases of CMIP can be achieved, and announce our intention to implement such a system for CMIP6. Accomplishing this could also free up valuable resources as many scientists are frequently "re-inventing the wheel" by re-writing analysis routines for well-established analysis methods. A more systematic approach for the community would be to develop and apply evaluation tools that are based on the latest scientific knowledge and observational reference, are well suited for routine use, and provide a wide range of diagnostics and performance metrics that comprehensively characterize model behaviour as soon as the output is published to the Earth System Grid Federation (ESGF). The CMIP infrastructure enforces data standards and conventions for model output and documentation accessible via the ESGF, additionally publishing observations (obs4MIPs) and reanalyses (ana4MIPs) for model intercomparison projects using the same data structure and organization as the ESM output. This largely facilitates routine evaluation of the ESMs, but to be able to process the data automatically alongside the ESGF, the infrastructure needs to be extended with processing capabilities at the ESGF data nodes where the evaluation tools can be executed on a routine basis. Efforts are already underway to develop community-based evaluation tools, and we encourage experts to provide additional diagnostic codes that would enhance this capability for CMIP. And, at the same time, we encourage the community to contribute observations and reanalyses for model evaluation to the obs4MIPs and ana4MIPs archives. The intention is to produce through the ESGF a widely accepted quasi-operational evaluation framework for CMIP6 that would routinely execute a series of standardized evaluation tasks. Over time, as this capability matures, we expect to produce an increasingly systematic characterization of models which, compared with early phases of CMIP, will more quickly and openly identify the strengths and weaknesses of the simulations. This will also reveal whether long-standing model errors remain evident in newer models and will assist modelling groups in improving their models. Finally, this framework will be designed to readily incorporate updates, including new observations and additional diagnostics and metrics as they become available from the research community.

54 ENVIRONMENTAL SCIENCES↗

The Fire and Smoke Model Evaluation Experiment—A Plan for Integrated, Large Fire–Atmosphere Field Campaigns

The Fire and Smoke Model Evaluation Experiment (FASMEE) is designed to collect integrated observations from large wildland fires and provide evaluation datasets for new models and operational systems. Wildland fire, smoke dispersion, and atmospheric chemistry models have become more sophisticated, and next-generation operational models will require evaluation datasets that are coordinated and comprehensive for their evaluation and advancement. Integrated measurements are required, including ground-based observations of fuels and fire behavior, estimates of fire-emitted heat and emissions fluxes, and observations of near-source micrometeorology, plume properties, smoke dispersion, and atmospheric chemistry. To address these requirements the FASMEE campaign design includes a study plan to guide the suite of required measurements in forested sites representative of many prescribed burning programs in the southeastern United States and increasingly common high-intensity fires in the western United States. Here we provide an overview of the proposed experiment and recommendations for key measurements. The FASMEE study provides a template for additional large-scale experimental campaigns to advance fire science and operational fire and smoke models.

54 ENVIRONMENTAL SCIENCES↗

Temperature and Humidity Profiles in the TqJoint Data Group of AIRS Version 6 Product for the Climate Model Evaluation

The Atmospheric Infrared Sounder (AIRS) mission is entering its 13th year of global observations of the atmospheric state, including temperature and humidity profiles, outgoing long-wave radiation, cloud properties, and trace gases. Thus AIRS data have been widely used, among other things, for short-term climate research and observational component for model evaluation. One instance is the fifth phase of the Coupled Model Intercomparison Project (CMIP5) which uses AIRS version 5 data in the climate model evaluation. The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) is the home of processing, archiving, and distribution services for data from the AIRS mission. The GES DISC, in collaboration with the AIRS Project, released data from the version 6 algorithm in early 2013. The new algorithm represents a significant improvement over previous versions in terms of greater stability, yield, and quality of products. The ongoing Earth System Grid for next generation climate model research project, a collaborative effort of GES DISC and NASA JPL, will bring temperature and humidity profiles from AIRS version 6. The AIRS version 6 product adds a new "TqJoint" data group, which contains data for a common set of observations across water vapor and temperature at all atmospheric levels and is suitable for climate process studies. How different may the monthly temperature and humidity profiles in "TqJoint" group be from the "Standard" group where temperature and water vapor are not always valid at the same time? This study aims to answer the question by comprehensively comparing the temperature and humidity profiles from the "TqJoint" group and the "Standard" group. The comparison includes mean differences at different levels globally and over land and ocean. We are also working on examining the sampling differences between the "TqJoint" and "Standard" group using MERRA data.

climate model evaluation↗

CrossCheck: Rapid, Reproducible, and Interpretable Model Evaluation

Evaluation beyond aggregate performance metrics, e.g. F1-score, is crucial to both establish an appropriate level of trust in machine learning models and identify future model improvements. In this paper we demonstrate CrossCheck, an interactive visualization tool for rapid crossmodel comparison and reproducible error analysis. We describe the tool and discuss design and implementation details. We then present three use cases (named entity recognition, reading comprehension, and clickbait detection) that show the benefits of using the tool for model evaluation. CrossCheck allows data scientists to make informed decisions to choose between multiple models, identify when the models are correct and for which examples, investigate whether the models are making the same mistakes as humans, evaluate models’ generalizability and highlight models’ limitations, strengths and weaknesses. Furthermore, CrossCheck is implemented as a Jupyter widget, which allows rapid and convenient integration into data scientists’ model development workflows.

Arendt, Dustin L.↗

Regime-oriented causal model evaluation of Atlantic–Pacific teleconnections in CMIP6

The climate system and its spatio-temporal changes are strongly affected by modes of long-term internal variability, like the Pacific decadal variability (PDV) and the Atlantic multidecadal variability (AMV). As they alternate between warm and cold phases, the interplay between PDV and AMV varies over decadal to multidecadal timescales. Here, we use a causal discovery method to derive fingerprints in the Atlantic–Pacific interactions and to investigate their phase-dependent changes. Dependent on the phases of PDV and AMV, different regimes with characteristic causal fingerprints are identified in reanalyses in a first step. In a second step, a regime-oriented causal model evaluation is performed to evaluate the ability of models participating in the Coupled Model Intercomparison Project Phase 6 (CMIP6) in representing the observed changing interactions between PDV, AMV and their extra-tropical teleconnections. The causal graphs obtained from reanalyses detect a direct opposite-sign response from AMV to PDV when analyzing the complete 1900–2014 period and during several defined regimes within that period, for example, when AMV is going through its negative (cold) phase. Reanalyses also demonstrate a same-sign response from PDV to AMV during the cold phase of PDV. Historical CMIP6 simulations exhibit varying skill in simulating the observed causal patterns. Generally, large-ensemble (LE) simulations showed better network similarity when PDV and AMV were out of phase compared to other regimes. Also, the two largest ensembles (in terms of number of members) were found to contain realizations with similar causal fingerprints to observations. For most regimes, these same models showed higher network similarity when compared to each other. This work shows how causal discovery on LEs complements the available diagnostics and statistical metrics of climate variability to provide a powerful tool for climate model evaluation.

54 ENVIRONMENTAL SCIENCES↗