Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model skill scores”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

A Numerical Study of Hypoxia in Chesapeake Bay Using an Unstructured Grid Model: Validation and Sensitivity to Bathymetry Representation

A three-dimensional unstructured-grid hydrodynamic and water quality model (Semi-implicit Cross-scale Hydroscience Intergrated System Model-Integrated Compartment Model) is applied successfully for Chesapeake Bay. The model is validated with observations of salinity, chlorophyll-a, dissolved oxygen, nutrients, and phytoplankton productions from the year 1991 to 1995 for the mainstem and some major tributaries, based on multiple model skill scores. Model experiments are conducted to test the importance of having (1) an accurate representation of bathymetry to correctly predict hypoxia and other processes and (2) a high-resolution model grid for tributaries to correctly simulate water quality variables. Further, comparison with the model experiment results with bathymetry smoothing indicates that bathymetry smoothing, as commonly used for many systems, changes the stratification and lateral circulation pattern, resulting in more salt intrusion into shallow water regions, and an increase in the freshwater age. Consequently, a model with bathymetry smoothing can lead to an unrealistic prediction of the distribution of hypoxia and phytoplankton production. Local grid refinement shows significant improvement of model simulations on local stratification and water quality variables. Overall, the use of high-resolution unstructured grid model leads to a faithful representation of the complex geometry, and thus a seamless cross-scale capability for simulating water quality processes in the Bay including tributaries and tidal creeks.

54 ENVIRONMENTAL SCIENCES↗

Alpha1 LASSO data bundles Lamont, OK

A data bundle is a unified package consisting of LASSO LES input and output, observations, evaluation diagnostics, and model skill scores. LES input includes model configuration information and forcing data. LES output includes profile statistics and full domain fields of cloud and environmental variables. Model evaluation data consists of LES output and ARM observations co-registered on the same grid and sampling frequency. Model performance is quantified by skill scores and diagnostics in terms of cloud and environmental variables.

54 ENVIRONMENTAL SCIENCES↗

LASSO Data Bundles

The Alpha 2 release is the second release from the LASSO Pilot Phase that builds upon the Alpha 1 release. Alpha 2 contains additional diagnostics in the data bundles and focuses on cases from spring-summer 2016. A data bundle is a unified package consisting of LASSO LES input and output, observations, evaluation diagnostics, and model skill scores. LES input include model configuration information and forcing data. LES output includes profile statistics and full domain fields of cloud and environmental variables. Model evaluation data consists of LES output and ARM observations co-registered on the same grid and sampling frequency. Model performance is quantified by skill scores and diagnostics in terms of cloud and environmental variables.

54 ENVIRONMENTAL SCIENCES↗

Assessing the Skill of the Improved Treatment of Riverine Freshwater in the Community Earth System Model (CESM) Relative to a New Salinity Climatology

Abstract Recent studies have explored the sensitivity of global ocean model simulations to the treatment of riverine freshwater and the representation of estuarine processes via an estuary box model applied within Community Earth System Model (CESM). This study builds on these efforts by assessing the model skill score relative to a new salinity climatology. The new climatology averages the original observational data of the World Ocean Database directly onto the CESM ocean component tracer grid cells without spatial interpolation, smoothing, or other gap‐filling techniques to mitigate coastal ocean salinity bias present in the World Ocean Atlas. The mean square error for coastal upper ocean salinity relative to climatology is reduced by up to 14%, and the mean square error of near‐surface salinity stratification is reduced by up to 28% near major river mouths in the simulations with improved treatments of river runoff. The improvement in upper ocean bulk salinity is attributed primarily to focusing runoff as point sources thereby avoiding the artificial horizontal spreading of the control run and to applying a locally varying instead of a global constant reference salinity for riverine virtual salt fluxes. The improvements in near‐surface salinity stratification are primarily attributed to adding parameterized estuarine mixing with the estuary box model. Salinity and salinity stratification skill improvements are achieved not just near large rivers but also along the global coast and skill improvements extend far offshore. Despite these improvements, many other sources of model‐climatology mismatch in coastal salinity and stratification remain and merit further attention.

58 GEOSCIENCES↗

Using ARM observations to evaluate NWP results of radiation and cloud fraction

This poster will present the results from: (1) Evaluating NWP model performances in simulating surface cloud radiative forcing measured by effective cloud albedo and total cloud fraction by comparing three major reanalysis datasets (ERAInterim, NCEP/NCAR Reanalysis and NCEP/DOE Reanalysis) against decade-long ARM observations; (2) Examining the relationships between the model biases of the cloud properties and those of nearsurface relative humidity and temperature; (3) Evaluating NWP model performances in simulating vertical profiles of cloud fraction; and, (4) Evaluating NWP model skill scores of predicting cloud fraction.

54 ENVIRONMENTAL SCIENCES↗

New framework for benchmarking decadal predictions leveraging the PCMDI Metric Package with interactive visualization

Reliable climate predictions across multiple timescales are increasingly critical as climate-related risks continue to rise. With the growing number and diversity of climate prediction systems, systematic intercomparison has become essential. Here, we present a comprehensive evaluation framework based on the PCMDI Metric Package to assess the performance of multiple decadal climate prediction systems. Unlike uninitialized simulations, initialized predictions exhibit bias and predictive skill that evolve with forecast lead time. To address this, we introduce (1) model-by-lead-time portrait plots, which efficiently summarize metrics of global temperature, precipitation, and Arctic/Antarctic sea-ice extent, and (2) an HTML-based interactive visualization platform that provides detailed regional and seasonal diagnostics of model bias, skill scores, and ensemble spread for each model and lead time. Comparisons with uninitialized simulations further quantify the relative impacts of initialization and external forcing on prediction skill. The proposed framework provides a scalable and transparent approach for multi-model climate prediction assessments and can be readily extended to a wide range of operational and research forecasting systems.

54 ENVIRONMENTAL SCIENCES↗

Convolutional neural networks for intra-hour solar forecasting based on sky image sequences

Accurate and timely solar forecasts play an increasingly critical role in power systems. Compared to longer forecasting timescales, very short-term solar forecasting has lagged behind in both research and practice. In this paper, we propose deep convolutional neural networks (CNNs) to provide operational intra-hour (10-minute-ahead to 60-minute-ahead) solar forecasts. We develop two CNN structures inspired by a widely-used CNN architecture. The CNNs are tailored to our solar forecasting regression tasks and rely solely on sky image sequences. Case studies based on six years of data (over 150,000 data points) demonstrate that the best CNN model has forecast skill scores of 20%-39% over the naive persistence of cloudiness benchmark, even at these very short timescales. The CNNs also have consistently superior performance when compared to shallow machine learning models with meteorological predictors, where the improvement averages around 7%. The sensitivity analyses show that the sky image length, resolution, and weather conditions have impacts on the deep learning model accuracy. In our intra-hour problem with specific setups, two sky images with a 10-minute 128 x 128 resolution yield the most accurate forecasts. Current limitations, future work, and deployment challenges and solutions are also discussed.

14 SOLAR ENERGY↗

Using Multiple Signatures to Improve Accuracy of Substorm Identification

Abstract We have developed a new procedure for combining lists of substorm onset times from multiple sources. We apply this procedure to observational data and to magnetohydrodynamic (MHD) model output from 1–31 January 2005. We show that this procedure is capable of rejecting false positive identifications and filling data gaps that appear in individual lists. The resulting combined onset lists produce a waiting time distribution that is comparable to previously published results, and superposed epoch analyses of the solar wind driving conditions and magnetospheric response during the resulting onset times are also comparable to previous results. Comparison of the substorm onset list from the MHD model to that obtained from observational data reveals that the MHD model reproduces many of the characteristic features of the observed substorms, in terms of solar wind driving, magnetospheric response, and waiting time distribution. Heidke skill scores show that the MHD model has statistically significant skill in predicting substorm onset times.

Haiducek, John D.↗

VALIDATING SOLAR WIND PREDICTION USING THE CURRENT SHEET SOURCE SURFACE MODEL

We have carried out a comparative study of the predicted solar wind based on the flux tube expansion factor computed using the current sheet source surface (CSSS) model and the potential field source surface (PFSS) model, with the aim of determining whether the CSSS model represents the solar wind sources better than the PFSS model. For this, we obtained the root mean square errors (RMSEs) and the correlation coefficients between the observed solar wind speed and that predicted by the models, the ratio of RMSEs between the two models, and a skill score. On average, the CSSS predictions are more accurate than the PFSS predictions by a factor of 1.6, taking RMSE as the metric of accuracy. The RMSEs increased as the solar cycle progressed toward maximum, indicating the difficulty in modeling the corona as the global field becomes more complex. We also compared the WSA/ENLIL predictions for a few Carrington rotations; the Wang-Sheeley-Arge (WSA) model makes use of the PFSS extrapolations to model the wind source. We found that the average value of RMSE ratio between the CSSS and the WSA/ENLIL predictions was about 1.9, implying that the CSSS predictions are nearly twice better than the WSA/ENLIL predictions, despite the simplicity of the CSSS model compared to ENLIL. We conclude, based on the present analysis, that the CSSS model is a valid proxy for solar wind measurements and that it improves upon existing commonly used methods of wind prediction or proxy analysis.

79 ASTRONOMY AND ASTROPHYSICS↗

Assessing probabilistic predictions of ENSO phase and intensity from the North American Multimodel Ensemble

Here we examine the skill of three, five, and seven-category monthly ENSO probability forecasts (1982–2015) from single and multi-model ensemble integrations of the North American Multimodel Ensemble (NMME) project. Three-category forecasts are typical and provide probabilities for the ENSO phase (El Niño, La Niña or neutral). Additional forecast categories indicate the likelihood of ENSO conditions being weak, moderate or strong. The level of skill observed for differing numbers of forecast categories can help to determine the appropriate degree of forecast precision. However, the dependence of the skill score itself on the number of forecast categories must be taken into account. For reliable forecasts with same quality, the ranked probability skill score (RPSS) is fairly insensitive to the number of categories, while the logarithmic skill score (LSS) is an information measure and increases as categories are added. The ignorance skill score decreases to zero as forecast categories are added, regardless of skill level. For all models, forecast formats and skill scores, the northern spring predictability barrier explains much of the dependence of skill on target month and forecast lead. RPSS values for monthly ENSO forecasts show little dependence on the number of categories. However, the LSS of multimodel ensemble forecasts with five and seven categories show statistically significant advantages over the three-category forecasts for the targets and leads that are least affected by the spring predictability barrier. These findings indicate that current prediction systems are capable of providing more detailed probabilistic forecasts of ENSO phase and amplitude than are typically provided.

54 ENVIRONMENTAL SCIENCES↗

Initial Results from the GEM Challenge on the Spacecraft Surface Charging Environment

Spacecraft surface charging during geomagnetically disturbed times is one of the most important causes of satellite anomalies. Predicting the surface charging environment is one prevalent task of the geospace environment models. Therefore, the GEM Focus Group “Inner Magnetosphere Cross-energy/Population Interactions” initiated a community-wide challenge study to assess the capability of several inner magnetosphere ring current models in determining surface charging environment for the Van Allen Probes orbits during the March 17, 2013 storm event. The integrated electron flux between 10 and 50 keV is used as the metrics. Various skill scores are applied to quantitatively measure the modeling performance against observations. Results indicate that no model consistently perform the best in all of the skill scores or for both satellites.We find that from these simulations the ring current model with observational flux boundary condition and Weimer electric potential driver generally reproduces the most realistic flux level around the spacecraft. A simple and weaker Volland-Stern electric field is not capable of effectively transporting the same plasma at the boundary towards the Earth. On the other hand, if the ring current model solves the electric field self-consistently and obtains similar strength and pattern in the equatorial plane as the Weimer model, the boundary condition plays another crucial role in determining the electron flux level in the inner region. Finally, when the boundary flux spectra based on MHD model/empirical model deviate from the shape or magnitude of the observed distribution function, the simulation produces poor skill scores along RBSP orbits.

54 ENVIRONMENTAL SCIENCES↗

Deterministic skill of ENSO predictions from the North American Multimodel Ensemble

Hindcasts and real-time predictions of the east-central tropical Pacific sea surface temperature (SST) from the North American Multimodel Ensemble (NMME) system are verified for 1982–2015. Skill is examined using two deterministic verification measures: mean squared error skill score (MSESS) and anomaly correlation. Verification of eight individual models shows somewhat differing skills among them, with some models consistently producing more successful predictions than others. The skill levels of MME predictions are approximately the same as the two best performing individual models, and sometimes exceed both of them. A decomposition of the MSESS indicates the presence of calibration errors in some of the models. In particular, the amplitudes of some model predictions are too high when predictability is limited by the northern spring ENSO predictability barrier and/or when the interannual variability of the SST is near its seasonal minimum. The skill of the NMME system is compared to that of the MME from the IRI/CPC ENSO prediction plume, both for a comparable hindcast period and also for a set of real-time predictions spanning 2002–2011. Comparisons are made both between the MME predictions of each model group, and between the average of the skills of the respective individual models in each group. Acknowledging a hindcast versus real-time inconsistency in the 2002–2012 skill comparison, the skill of the NMME is slightly higher than that of the prediction plume models in all cases. This result reflects well on the NMME system, with its large total ensemble size and opportunity for possible complementary contributions to skill.

54 ENVIRONMENTAL SCIENCES↗

Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction

Abstract Background West Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental USA. WNV occurrence has high spatiotemporal variation, and current approaches to targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement. Methods We performed a multi-model comparative assessment of probabilistic forecasts submitted by 15 teams for annual WNND cases in US counties for 2020 and assessed forecast accuracy, calibration, and discriminatory power. In the evaluation, we included forecasts produced by comparison models of varying complexity as benchmarks of forecast performance. We also used regression analysis to identify modeling approaches and contextual factors that were associated with forecast skill. Results Simple models based on historical WNND cases generally scored better than more complex models and combined higher discriminatory power with better calibration of uncertainty. Forecast skill improved across updated forecast submissions submitted during the 2020 season. Among models using additional data, inclusion of climate or human demographic data was associated with higher skill, while inclusion of mosquito or land use data was associated with lower skill. We also identified population size, extreme minimum winter temperature, and interannual variation in WNND cases as county-level characteristics associated with variation in forecast skill. Conclusions Historical WNND cases were strong predictors of future cases with minimal increase in skill achieved by models that included other factors. Although opportunities might exist to specifically improve predictions for areas with large populations and low or high winter temperatures, areas with high case-count variability are intrinsically more difficult to predict. Also, the prediction of outbreaks, which are outliers relative to typical case numbers, remains difficult. Further improvements to prediction could be obtained with improved calibration of forecast uncertainty and access to real-time data streams (e.g. current weather and preliminary human cases). Graphical Abstract

54 ENVIRONMENTAL SCIENCES↗

A new method for predicting hurricane rapid intensification based on co-occurring environmental parameters

Abstract Tropical cyclones (TCs) that undergo Rapid Intensification (RI) can pose serious socioeconomic threats and can potentially result in major damaging impacts along coastal areas. Considering the complexity of various physical mechanisms that play a role in RI and its relatively low probability of occurrence, predicting RI remains a major operational challenge. In this study, we propose a simple deterministic binary classification model based on the co-occurrence of environmental parameters (MCE) to predict an RI event. More specifically, the model determines the possibility of RI based on a simple count of the number of environmental predictors deemed favorable and unfavorable. We compare our model results to logistic regression (LR) and decision tree (DT) models, well-trained using the same set of environmental predictors. Results reveal that at an RI threshold of 30 kt, the MCE exhibits a critical success index score of 0.233 which is 14% higher than DT and LR model performances. When tested at multiple RI thresholds, the MCE displays relatively higher skill scores across multiple metrics. By simultaneously evaluating the favorability of predictors, the MCE is able to comparatively reduce the number of false alarms predicted when certain predictors are unfavorable toward RI. Interpreting these model results to gain a physical understanding of how co-occurring environmental parameters can affect RI, we highlight future directions for using models based on the MCE approach to understand and predict TC RI as well as other meteorological extremes.

54 ENVIRONMENTAL SCIENCES↗

Evaluation of Subseasonal-to-Seasonal (S2S) precipitation forecast from the North American Multi-Model ensemble phase II (NMME-2) over the contiguous U.S.

The second phase of the North America Multi-Model Ensemble (NMME-2) provides globally available Subseasonal-to-Seasonal (S2S) precipitation forecasts with a daily resolution. The S2S precipitation forecasts are getting increasing attention for their potentials in providing hydrometeorological forcing information for water resources planning at an extended range. However, the forecast skills of many existing S2S forecast products will significantly decrease when the lead time increases, hindering their applicability for watershed-scale hydrologic modeling. Therefore, forecast validation and large-scale evaluation are of great importance for water resources planning and hydrological applications. In this study, we comprehensively evaluate the S2S precipitation forecasts from the NMME-2 dataset over the contiguous United States (CONUS) and during the study period from 1982 to 2011. Three aspects of precipitation forecast capabilities are compared and analyzed: bias, skill scores, and the ability to predict extreme precipitation events. The Parameter-elevation Regressions on Independent Slopes Model (PRISM) is used as ground truth reference. Differs from other regional forecast validation study, we further examined and analyzed the dependences of NMME-2 precipitation forecast skills according to different seasonality, geographical locations, and lead times. Results show that the forecast biases are not sensitive to lead times but are seasonally dependent of all NMME-2 models. Overestimations are found in the Western U.S. in cooler seasons while underestimations are observed in the central regions of the U.S. in warmer seasons. The forecast skill of all individual NMME-2 models generally decreases as increases of lead times. The simple model averaging (SMA) of five NMME-2 models demonstrates a higher forecast skill than any individual NMME-2 models. Spatially, the highest forecast skill scores are observed at coastal areas in the Western U.S. with an one-week lead time. As compared to the historical resampled forecasts, NMME-2 also shows better performance in predicting extreme precipitation events above 99% percentiles and below 1% percentiles with higher probability of detections and lower false alarm ratios. Finally, the obtained results suggest the great potentials of NMME-2 precipitation forecasts in assisting ensemble hydrologic forecasts at the S2S scale over the CONUS.

54 ENVIRONMENTAL SCIENCES↗

Description of the LASSO Data Bundles Product

The U. S. Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) user facility began a pilot project in May 2015 to design a routine, high-resolution modeling capability to complement ARM’s extensive suite of measurements. This modeling capability has evolved into the Large-Eddy Simulation (LES) ARM Symbiotic Simulation and Observation (LASSO) datastream. The datastream, broadly termed data bundles, contains high-resolution model output, input files, observations for evaluation, and skill scores for the simulations. The initial focus of LASSO is on shallow convection at the ARM Southern Great Plains (SGP) atmospheric observatory. The availability of LES simulations with concurrent observations serves many purposes. LES helps bridge the scale gap between DOE ARM observations and models, and the use of routine LES adds value to observations. It provides a self-consistent representation of the atmosphere and a dynamical context for the observations. Further, it elucidates unobservable processes and properties. LASSO generates a simulation library for researchers that enables statistical approaches beyond a single-case mentality. It also provides tools necessary for modelers to reproduce the LES and conduct their own sensitivity experiments. The LASSO library of data bundles is designed to facilitate a wide range of research. For an observationalist, LASSO can help inform instrument remote-sensing retrievals, conduct observation system simulation experiments (OSSEs), and test implications of radar scan strategies or flight paths. For a theoretician, LASSO can help calculate estimates of fluxes and co-variability of values, and test relationships without having to run the model personally. For a modeler, LASSO can help one know ahead of time which days have good forcing, have co-registered observations at high-resolution scales, and have simulation inputs and corresponding outputs to test parameterizations. Further details on the overall LASSO project are available at https://www.arm.gov/capabilities/modeling/lasso.

54 ENVIRONMENTAL SCIENCES↗

Model evaluation guidelines for geomagnetic index predictions

Geomagnetic indices are convenient quantities that distill the complicated physics of some region or aspect of near–Earth space into a single parameter. Most of the best–known indices are calculated from ground–based magnetometer data sets, such as Dst, SYM–H, Kp, AE, AL, and PC. Many models have been created that predict the values of these indices, often using solar wind measurements upstream from Earth as the input variables to the calculation. This document reviews the current state of models that predict geomagnetic indices and the methods used to assess their ability to reproduce the target index time series. These existing methods are synthesized into a baseline collection of metrics for benchmarking a new or updated geomagnetic index prediction model. These methods fall into two categories: (1) fit performance metrics such as root mean square error (RMSE) and mean absolute error (MAE) that are applied to a time–series comparison of model output and observations; and (2) event detection performance metrics such as Heidke Skill Score and probability of detection (POD) that are derived from a contingency table that compares model and observation values exceeding (or not) a threshold value. Furthermore, a few examples of codes being used with this set of metrics are presented, and other aspects of metrics assessment best practices, limitations, and uncertainties are discussed, including several caveats to consider when using geomagnetic indices.

79 ASTRONOMY AND ASTROPHYSICS↗