Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “benchmarks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,729 records · Page 96

smol_model_benchmark

Machine learning models for protein–small molecule binding prediction using graph (GAT), image-based (ResNet), and transformer-based SMILES approaches (ChemBERTa and MIST). The code includes training pipelines and experiments on the dataset from the BELKA Kaggle competition.

Gibson, Kaetlyn [Los Alamos National Lab]↗

Benchmarking Simulated Precipitation in Earth System Models

Earth system models (ESMs) bridge observationally based and theoretical understanding of the Earth system. They are among the most often used tools to study a variety of questions related to variability and changes in Earth’s climate. For many applications, ESMs must realistically simulate observed large-scale precipitation patterns and seasonal cycles that have a multitude of societal and national security implications. Despite steady improvement in the simulation of precipitation, model errors in many aspects of precipitation characteristics limit the use of ESMs both in understanding Earth system variability and change and for decision-making.

54 ENVIRONMENTAL SCIENCES↗

Benchmarking Near-Surface Winds in the HRRR Analyses Using Multisource Observations over Complex Terrain in the Southeastern United States

Wind energy plays a crucial role in sustainable power generation, yet its full potential in the southeast United States (SEUS) remains underexplored. This study advances wind resource assessment in the SEUS using existing regional modeling and multisource observations. We find that terrain complexity, influenced by orography and forest canopy, significantly impacts the accuracy of modeled wind speed. While the High-Resolution Rapid Refresh (HRRR) model effectively simulates wind profiles over flat, nonforested terrain, larger errors are produced above forest canopies, and particularly for those stands in hilly and mountainous terrain where there are no observations aloft that could be used by HRRR’s data assimilation software. Seasonal variations, e.g., changes in the leaf area index, further complicate the relationship between terrain complexity and simulation errors. Here, our findings emphasize the critical need for comprehensive wind profile measurements, extending from below the canopy to height above the canopy and through the lower planetary boundary layer, to fully quantify model performance over and near forests, and for improving wind resource assessment, planning, and management. This study provides valuable insights not only for further model development but also for future field campaign deployment with the goal of further improving our understanding of wind resources in the region.

Complex terrain↗

Metrics for the Diurnal Cycle of Precipitation: Toward Routine Benchmarks for Climate Models

In this paper, metrics are proposed—that is, a few summary statistics that condense large amounts of data from observations or model simulations—encapsulating the diurnal cycle of precipitation. Vector area averaging of Fourier amplitude and phase produces useful information in a reasonably small number of harmonic dial plots, a procedure familiar from atmospheric tide research. The metrics cover most of the globe but down-weight high-latitude wintertime ocean areas where baroclinic waves are most prominent. This enables intercomparison of a large number of climate models with observations and with each other. The diurnal cycle of precipitation has features not encountered in typical climate model intercomparisons, notably the absence of meaningful “average model” results that can be displayed in a single two-dimensional map. Displaying one map per model guides development of the metrics proposed here by making it clear that land and ocean areas must be averaged separately, but interpreting maps from all models becomes problematic as the size of a multimodel ensemble increases. Global diurnal metrics provide quick comparisons with observations and among models, using the most recent version of the Coupled Model Intercomparison Project (CMIP). This includes, for the first time in CMIP, spatial resolutions comparable to global satellite observations. Finally, consistent with earlier studies of resolution versus parameterization of the diurnal cycle, the longstanding tendency of models to produce rainfall too early in the day persists in the high-resolution simulations, as expected if the error is due to subgrid-scale physics.

54 ENVIRONMENTAL SCIENCES↗

Benchmarking performance changes in the simulation of extratropical modes of variability across CMIP generations

In this work, we evaluate extratropical modes of variability in the three most recent phases of the Coupled Model Intercomparison Project (CMIP3, 5, and 6) to gauge improvement of climate models over time. A suite of high-level metrics is employed to objectively evaluate how well climate models simulate the qobserved Northern Annular Mode (NAM), North Atlantic Oscillation (NAO), Pacific North America pattern (PNA), Southern Annular Mode (SAM), Pacific Decadal Oscillation (PDO), North Pacific Oscillation (NPO), and North Pacific Gyre Oscillation (NPGO). We apply a Common Basis Function (CBF) approach that projects model anomalies onto observed empirical orthogonal functions (EOFs), together with the traditional EOF approach, to CMIP Historical and AMIP models. We find simulated spatial patterns of those modes have been significantly improved in the newer models, although the skill improvement is sensitive to the mode and season considered. We identify some potential contributions to the pattern improvement of certain modes (e.g., the Southern Hemisphere jet and High-top vertical coordinate), however the performance changes are likely attributed to gradual improvement of the base climate and multiple relevant processes. Less performance improvement is evident in the mode amplitude of these modes and systematic overestimation of the mode amplitude in spring remains in the newer climate models. We find that the post-dominant season amplitude errors in atmospheric modes are not limited to coupled runs but are often already evident in AMIP simulations. This suggests that rectifying the egregious post-dominant season amplitude errors found in many models can be addressed in an atmospheric-only framework making it more tractable to address in the model development process.

54 ENVIRONMENTAL SCIENCES↗

Benchmarking Simulated Precipitation Variability Amplitude across Time Scales

Here, objective performance metrics that measure precipitation variability across time scales from subdaily to interannual are presented and applied to Historical simulations of Coupled Model Intercomparison Project phase 5 and 6 (CMIP5 and CMIP6) models. Three satellite-based precipitation estimates (IMERG, TRMM, and CMORPH) are used as reference data. We apply two independent methods to estimate temporal variability of precipitation and compare the consistency in their results. The first method is derived from power spectra analysis of 3-hourly precipitation, measuring forced variability by solar insolation (diurnal and annual cycles) and internal variability at different time scales (subdaily, synoptic, subseasonal, seasonal, and interannual). The second method is based on time averaging and facilitates estimating the seasonality of subdaily variability. Supporting the robustness of our metric, we find a near equivalence between the results obtained from the two methods when examining simulated-to-observed ratios over large domains (global, tropics, extratropics, land, or ocean). Additionally, we demonstrate that our model evaluation is not very sensitive to the discrepancies between observations. Our results reveal that CMIP5 and CMIP6 models in general overestimate the forced variability while they underestimate the internal variability, especially in the tropical ocean and higher-frequency variability. The underestimation of subdaily variability is consistent across different seasons. The internal variability is overall improved in CMIP6, but remains underestimated, and there is little evidence of improvement in forced variability. Increased horizontal resolution results in some improvement of internal variability at subdaily and synoptic time scales, but not at longer time scales.

54 ENVIRONMENTAL SCIENCES↗

Benchmarks and Tests of a Multidimensional Cluster Dynamics Model of Helium Implantation in Tungsten

Here, we present a hierarchical multiscale modeling study of implanted helium (He) segregation near grain boundaries (GBs) of tungsten. We extend our spatially dependent cluster dynamics model to two spatial dimensions in order to take into account the biased drift of mobile He clusters toward the GBs observed in atomic-scale simulations. We are also able to reproduce the results from large-scale molecular dynamics simulations near and away from the GBs at low fluence with the extended cluster dynamics model. We suggest and verify that the sink (surface and GB) strengths are attenuated by the increasing concentration of He clusters at high fluence. This cluster dynamics model continues to set the stage for development of fully atomistically informed, coarse-grained models for computationally efficient predictions of He retention and surface morphological evolution, advancing progress toward the goal of efficient and optimal design of plasma-facing components.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Bayesian Monte Carlo Evaluation Framework for Cross Sections Nuclear Data and Integral Benchmark Experiments

The new Bayesian Monte Carlo (MC) evaluation framework described in this abstract has been conceived as an attempt to improve nuclear data evaluations of differential crosssection data by removing the following two approximations conventionally employed for nuclear data evaluations: all probability density functions (PDFs) of all data and model parameters, both prior and posterior, are assumed to be normal (i.e., Gaussian) PDFs, and all uncertainties and covariances are propagated using a linear approximation. With these approximations removed, the Bayesian MC (BMC) framework could be used to account for nonlinear effects and would enable improved evaluations of differential cross sections and IBE data that are presently performed based on the assumptions itemized above. The BMC would also improve upon the uniform sampling of IBE parameters from within ranges defined by their evaluated uncertainties.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗