Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 469 records · Page 26

A curated benchmark for cofolding models on kinase conformational states

Abstract Protein kinases are critical drug targets, requiring therapeutics that can modulate their active and inactive conformational states. While cofolding models can generate global folds directly from kinase sequences and ligand SMILES strings, these models have not yet been tested on their ability to recover ligand-induced-fit conformational states of the kinase proteins. Here, we introduce KinConfBench, a curated benchmark of 2225 high-quality human kinase chains to evaluate the ability of four state-of-the-art cofolding models—Boltz-2, Chai-1, Protenix, and RoseTTAFold-All-Atom—to recover both canonical and rare conformational states. We show that geometric success metrics of a ligand pose in the active site do not correlate strongly with the correct kinase conformational state, motivating a new set of dynamical benchmarks for assessing cofolding models. While all four cofolding models achieve ~60–80% prediction accuracy for kinase conformational classification, they exhibit severe mode collapse when performing multiple inferences, show negligible structural diversity in sampling induced-fit motions, and display a prevalent “apo-drift” in which most cofolding models predominantly predict the kinase to be in its ligand-free state. Our results highlight that capturing ligand-induced protein conformational diversity, not just geometric fit, is critical for next-generation structure-based drug discovery.

Sun, Kunyang↗

wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation

As machine learning (ML) is increasingly implemented in hardware to address real-time challenges in scientific applications, the development of advanced toolchains has significantly reduced the time required to iterate on various designs. These advancements have solved major obstacles, but also exposed new challenges. For example, processes that were not previously considered bottlenecks, such as hardware synthesis, are becoming limiting factors in the rapid iteration of designs. To mitigate these emerging constraints, multipleefforts have been undertaken to develop an ML-based surrogate model that estimates resource usage of synthesized ML accelerator architectures. We introduce wa-hls4ml, a benchmark for ML accelerator resource and latency estimation, and its corresponding initial dataset of over 680 000 fully connected and convolutional neural networks, all synthesized using hls4ml and targeting Xilinx FPGAs. The benchmark evaluates the performance of resource and latency predictors against several common ML model architectures, primarily originating from scientific domains, as exemplar models, and the average performance across a subset of the dataset. Additionally, we introduce GNN- and transformer-based surrogate models that predict latency and resources for ML accelerators. We present the architecture and performance of the models and find that the models generally predict latency and resources for the 75% percentile within several percent of the synthesized resources on the synthetic test dataset.

Hawks, Benjamin G. [Fermilab]↗

Four-Nozzle Benchmark Wind Tunnel Model USA Code Solutions for Simulation of Multiple Rocket Base Flow Recirculation at 145,000 Feet Altitude

Multiple rocket exhaust plume interactions at high altitudes can produce base flow recirculation with attendant alteration of the base pressure coefficient and increased base heating. A search for a good wind tunnel benchmark problem to check grid clustering technique and turbulence modeling turned up the experiment done at AEDC in 1961 by Goethert and Matz on a 4.25-in. diameter domed missile base model with four rocket nozzles. This wind tunnel model with varied external bleed air flow for the base flow wake produced measured p/p(sub ref) at the center of the base as high as 3.3 due to plume flow recirculation back onto the base. At that time in 1961, relatively inexpensive experimentation with air at gamma = 1.4 and nozzle A(sub e)/A of 10.6 and theta(sub n) = 7.55 deg with P(sub c) = 155 psia simulated a LO2/LH2 rocket exhaust plume with gamma = 1.20, A(sub e)/A of 78 and P(sub c) about 1,000 psia. An array of base pressure taps on the aft dome gave a clear measurement of the plume recirculation effects at p(infinity) = 4.76 psfa corresponding to 145,000 ft altitude. Our CFD computations of the flow field with direct comparison of computed-versus-measured base pressure distribution (across the dome) provide detailed information on velocities and particle traces as well eddy viscosity in the base and nozzle region. The solution was obtained using a six-zone mesh with 284,000 grid points for one quadrant taking advantage of symmetry. Results are compared using a zero-equation algebraic and a one-equation pointwise R(sub t) turbulence model (work in progress). Good agreement with the experimental pressure data was obtained with both; and this benchmark showed the importance of: (1) proper grid clustering and (2) proper choice of turbulence modeling for rocket plume problems/recirculation at high altitude.

Dougherty, N. S.↗

Cooling Matters: Benchmarking Large Language Models and Vision-Language Models on Liquid-Cooled Versus Air-Cooled H100 GPU Systems

The unprecedented growth in artificial intelligence (AI) workloads, recently dominated by large language models (LLMs) and vision-language models (VLMs), has intensified power and cooling demands in data centers. This study benchmarks LLMs and VLMs on two HGX nodes, each with 8× NVIDIA H100 graphics processing units (GPUs), using liquid and air cooling. Leveraging GPU Burn, Weights & Biases, and IPMItool, we collect detailed thermal, power, and computation data. Results show that the liquid-cooled systems maintain GPU temperatures between 41-50$^\circ$C, while the air-cooled counterparts fluctuate between 54-72$^\circ$C under load. This thermal stability of liquid-cooled systems yields 17% higher performance (54 TFLOPs/ GPU vs. 46 TFLOPs/GPU), performance-per-watt, reduced energy overhead, and greater system efficiency than the air-cooled counterparts. These findings underscore the energy and sustainability benefits of liquid cooling, offering a compelling path forward for hyperscale data centers seeking to optimize AI infrastructure. https://github.com/iscaas/Cooling-Matters.

Latif, Imran↗

The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider

We describe the outcome of a data challenge conducted as part of the Dark Machines (https://www.darkmachines.org) initiative and the Les Houches 2019 workshop on Physics at TeV colliders. The challenged aims to detect signals of new physics at the Large Hadron Collider (LHC) using unsupervised machine learning algorithms. First, we propose how an anomaly score could be implemented to define model-independent signal regions in LHC searches. We define and describe a large benchmark dataset, consisting of >1 billion simulated LHC events corresponding to 10\, fb^{-1} 10 f b − 1 of proton-proton collisions at a center-of-mass energy of 13 TeV. We then review a wide range of anomaly detection and density estimation algorithms, developed in the context of the data challenge, and we measure their performance in a set of realistic analysis environments. We draw a number of useful conclusions that will aid the development of unsupervised new physics searches during the third run of the LHC, and provide our benchmark dataset for future studies at https://www.phenoMLdata.org. Code to reproduce the analysis is provided at https://github.com/bostdiek/DarkMachines-UnsupervisedChallenge.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Cosmogenic radionuclide production modeling with Geant4: Experimental benchmarking and application to nuclear spectroscopy of asteroid (16) Psyche

Measurements of gamma-ray emissions from the decay of cosmogenic radionuclides provide an opportunity to characterize the elemental composition of a terrestrial planet or asteroid surface. We report on the development of a Geant4 application which models cosmogenic radionuclide production on metal-rich surfaces. The model was benchmarked using measurements of radionuclides produced during 1 GeV proton irradiation of a target made from the Campo del Cielo iron meteorite. The gamma-ray emitting radionuclides 58 Co, 57 Co, 56 Co, 54 Mn, 52 Mn, 51 Cr, 48 V, 46 Sc and 22 Na were observed following the irradiation. Our model reproduced the measured radionuclide production rates to within a factor of 2.5 or better. All but two of the elements ( 54 Mn, 46 Sc) have a perfect model-to-data match within their measurement uncertainties. The benchmarked model was used to predict cosmogenic radionuclide production on a large (~100-km radius) metal-rich asteroid. The results are relevant for planned gamma-ray measurements of the metallic asteroid (16) Psyche, which will be visited by the Psyche spacecraft in 2026. We found that galactic-proton-induced radionuclide decay is unlikely to be observed by the Psyche Gamma-Ray Spectrometer (GRS), however an intense solar proton event (>2×10 6 protons cm -2 over <5 days) will produce measurable quantities of radioisotopes. Measurements of these radioisotopes could provide an independent method of determining the Ni-to-Fe ratio of materials at Psyche’s surface. Such an analysis will require the use of radionuclide production cross sections to convert GRS-measured cosmogenic radionuclide decay rates to elemental composition information with the ~10% precision required for planetary geochemical studies.

79 ASTRONOMY AND ASTROPHYSICS↗

Evaluating three evapotranspiration estimates from model of different complexity over China using the ILAMB benchmarking system

The land surface models range in complexity of terrestrial evapotranspiration, yet it is unknown how model complexity translates to accuracy of modeled evapotranspiration estimates. Here, we use the International Land Model Benchmarking system to assess ET estimates from three models of varying complexity driven by the same forcing datasets: an earth system model, a terrestrial biosphere model, and a stand-alone ET model. The performance assessment includes both temporal and spatial evaluation, and different plant functional types across China. Our results indicate that the most complex model, an earth system model, performed best against the benchmarking datasets and metrics. Terrestrial biosphere model performed best in simulating inter-annual variability of ET, while earth system model performed best in simulating the seasonal cycle. The more complex models (earth system model and terrestrial biosphere model) perform better in forest, shrub and crop ecosystems, while the simpler model (stand-alone ET model) perform better in grass ecosystems. Our study demonstrates the impact of model complexity on ET estimates and highlights directions for future ET model improvements.

54 ENVIRONMENTAL SCIENCES↗

3D CFD Model Validation Using Benchmark Data of 1/16th Scaled VHTR Upper Plenum and Development of Wall Heat-Transfer Correlation For Laminar Flow

With support from the U.S. Department of Energy-Office of Nuclear Energy’s (DOE-NE’s) Nuclear Energy Advanced Modeling and Simulation (NEAMS) program, an effort has been pursued to support high-temperature gas-cooled reactor (HTGR) technology development and its modeling and simulation needs. There is a particular need for advanced modeling and simulation tools to predict thermal-fluid behavior in the nuclear reactor primary system, especially in the core and the lower and upper plena, during safety-related transients. In this report, two main such activities are presented relevant to the HTGRs: (1) three-dimensional (3D) computational fluid dynamics (CFD) validation using benchmark data from the upper plenum of Texas A&M University’s 1/16th scaled very-high-temperature gas-cooled reactor (VHTR), and (2) development of wall heat-transfer correlation for laminar flow in a wall-heated pipe. The CFD tool validation exercises can be helpful to choose the models and CFD tools to simulate and design specific components of the HTRGs such as upper plenum where jet mixing is a complex phenomenon. In a loss of forced circulation event, the laminar flow can be observed during the development of natural circulation flow. This work includes the development and validation of heat transfer correlations for laminar flow using the Nek5000 CFD code due to limited available experimental data for laminar flow conditions to guide low-order models (1D). In this report, the flow characteristics of a single isothermal jet discharging into the upper plenum was investigated using the Nek5000 Large-Eddy Simulation (LES) CFD tool. Several numerical simulations were performed for various jet-discharged Reynolds numbers ranging from 3,413 to 12,819. A grid-independent study was performed. The numerical results of mean velocity, root-mean-square fluctuating velocity, and Reynolds stress were compared against the benchmark data. Good agreement was obtained between simulated and measured data for axial mean velocities, except near the upper plenum hemisphere. The maximum predicted errors for axial mean velocities at various normalized coolant channel diameter heights of 1, 5, and 10 are 1.56%, 1.88%, and 3.82%, respectively. In addition, the predicted root-mean-square fluctuating velocity and Reynolds stress are qualitatively in agreement with the experimental data. The Nek5000 code was used to develop wall-heat transfer correlation for laminar flow in a cylindrical tube. Several simulations were performed for various Reynolds flow and wall-heat fluxes. A new heat transfer correlation was developed using data from Nek5000 simulation results and regression functions in Matlab. The developed heat transfer correlation is valid for various Reynolds flows from 200 to 2000. The predicted R² value for model fit was 0.875, which ensures that 87.5% of the model data lies on the Nek5000 data. Moreover, a machine learning (ML) tool was used to train and test the Nek5000 data. A good fit of the ML-based model was observed with the test data.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Generation IV benchmarking of TRISO fuel performance models under accident conditions: Modeling input data

This document presents the benchmark plan for the calculation of particle fuel performance on safety testing experiments that are representative of operational accidental transients. The benchmark is dedicated to the modeling of fission product release under accident conditions by fuel performance codes from around the world, and the subsequent comparison to post-irradiation experiment (PIE) data from the modeled heating tests. The accident condition benchmark is divided into three parts: the modeling of a simplified benchmark problem to assess potential numerical calculation issues at low fission product release; the modeling of the AGR-1 and HFR-EU1bis safety testing experiments; and, the comparison of the AGR-1 and HFR-EU1bis modeling results with PIE data. The simplified benchmark case, thereafter named NCC (Numerical Calculation Case), is derived from ''Case 5'' of the International Atomic Energy Agency (IAEA) Coordinated Research Program (CRP) on coated particle fuel technology [IAEA 2012]. It is included so participants can evaluate their codes at low fission product release. ''Case 5'' of the IAEA CRP-6 showed large code-to-code discrepancies in the release of fission products, which were attributed to ''effects of the numerical calculation method rather than the physical model''[IAEA 2012]. The NCC is therefore intended to check if these numerical effects subsist. The first two steps imply the involvement of the benchmark participants with a modeling effort following the guidelines and recommendations provided by this document. The third step involves the collection of the modeling results by Idaho National Laboratory (INL) and the comparison of these results with the available PIE data. The objective of this document is to provide all necessary input data to model the benchmark cases, and to give some methodology guidelines and recommendations in order to make all results suitable for comparison with each other. The participants should read this document thoroughly to make sure all the data needed for their calculations is provided in the document. Missing data will be added to a revision of the document if necessary.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

GEN-IV Benchmarking of Triso Fuel Performance Models under accident conditions modeling input data

This document presents the benchmark plan for the calculation of particle fuel performance on safety testing experiments that are representative of operational accidental transients. The benchmark is dedicated to the modeling of fission product release under accident conditions by fuel performance codes from around the world, and the subsequent comparison to post-irradiation experiment (PIE) data from the modeled heating tests. The accident condition benchmark is divided into three parts: • The modeling of a simplified benchmark problem to assess potential numerical calculation issues at low fission product release. • The modeling of the AGR-1 and HFR-EU1bis safety testing experiments. • The comparison of the AGR-1 and HFR-EU1bis modeling results with PIE data. The simplified benchmark case, thereafter named NCC (Numerical Calculation Case), is derived from “Case 5” of the International Atomic Energy Agency (IAEA) Coordinated Research Program (CRP) on coated particle fuel technology [IAEA 2012]. It is included so participants can evaluate their codes at low fission product release. “Case 5” of the IAEA CRP-6 showed large code-to-code discrepancies in the release of fission products, which were attributed to “effects of the numerical calculation method rather than the physical model” [IAEA 2012]. The NCC is therefore intended to check if these numerical effects subsist. The first two steps imply the involvement of the benchmark participants with a modeling effort following the guidelines and recommendations provided by this document. The third step involves the collection of the modeling results by Idaho National Laboratory (INL) and the comparison of these results with the available PIE data. The objective of this document is to provide all necessary input data to model the benchmark cases, and to give some methodology guidelines and recommendations in order to make all results suitable for comparison with each other. The participants should read this document thoroughly to make sure all the data needed for their calculations is provided in the document. Missing data will be added to a revision of the document if necessary. 09/2016: Tables 6 and 8 updated. AGR-2 input data added

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Benchmarking the exponential ansatz for the Holstein model

Polarons are quasiparticles formed as a result of lattice distortions induced by charge carriers. The single-electron Holstein model captures the fundamentals of single polaron physics. We examine the power of the exponential ansatz for the polaron ground-state wavefunction in its coupled cluster, canonical transformation, and (canonically transformed) perturbative variants across the parameter space of the Holstein model. Our benchmark serves to guide future developments of polaron wavefunctions beyond the single-electron Holstein model.

Chemistry↗

A Segmentation Algorithm for Characterizing Rise and Fall Segments in Seasonal Cycles: An Application to XCO2 to Estimate Benchmarks and Assess Model Bias

There is more useful information in the time series of satellite-derived column-averaged carbon dioxide (XCO2) than is typically characterized. Often, the entire time series is treated at once without considering detailed features at shorter timescales, such as nonstationary changes in signal characteristics – amplitude, period and phase. In many instances, signals are visually and analytically differentiable from other portions in a time series. Each rise (increasing) and fall (decreasing) segment in the seasonal cycle is visually discernable in a graph of the time series. The rise and fall segments largely result from seasonal differences in terrestrial ecosystem production, which means that the segment's signal characteristics can be used to establish observational benchmarks because the signal characteristics are driven by similar underlying processes. We developed an analytical segmentation algorithm to characterize the rise and fall segments in XCO2 seasonal cycles. We present the algorithm for general application of the segmentation analysis and emphasize here that the segmentation analysis is more generally applicable to cyclic time series. We demonstrate the utility of the algorithm with specific results related to the comparison between satellite- and model-derived XCO2 seasonal cycles (2009–2012) for large bioregions across the globe. We found a seasonal amplitude gradient of 0.74–0.77 ppm for every 10∘ of latitude in the satellite data, with similar gradients for rise and fall segments. This translates to a south–north seasonal amplitude gradient of 8 ppm for XCO2, about half the gradient in seasonal amplitude based on surface site in situ CO2 data (∼19 ppm). The latitudinal gradients in the period of the satellite-derived seasonal cycles were of opposing sign and magnitude (−9 d per 10∘ latitude for fall segments and 10 d per 10∘ latitude for rise segments) and suggest that a specific latitude (∼2∘ N) exists that defines an inversion point for the period asymmetry. Before (after) the point of asymmetry inversion, the periods of rise segments are lesser (greater) than the periods of fall segments; only a single model could reproduce this emergent pattern. The asymmetry in amplitude and the period between rise and fall segments introduces a novel pattern in seasonal cycle analyses, but, while we show these emergent patterns exist in the data, we are still breaking ground in applying the information for science applications. Maybe the most useful application is that the segmentation analysis allowed us to decompose the model biases into their correlated parts of biases in amplitude, period and phase independently for rise and fall segments. We offer an extended discussion on how such information about model biases and the emergent patterns in satellite-derived seasonal cycles can be used to guide future inquiry and model development.

Calle, Leonardo↗

Low frequency eddy current benchmark study for model validation

This paper presents results of an eddy current model validation study. Precise measurements were made using an impedance analyzer to investigate changes in impedance due to Electrical Discharge Machining (EDM) notches in aluminum plates. Each plate contained one EDM notch at an angle of 0, 10, 20, or 30 degrees from the normal of the plate surface. Measurements were made with the eddy current probe both scanning parallel and perpendicular to the notch length. The experimental response from the vertical and oblique notches will be reported and compared to results from different numerical simulation codes.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

A proposal for the intercomparison of the dynamical cores of atmospheric general circulation models

A benchmark calculation is proposed for evaluating the dynamical cores of atmospheric general circulation models (GCMs) independently of the physical parameterizations. The test focuses on the long-term statistical properties of a fully developed general circulation; thus, it is particularly appropriate for intercomparing the dynamics used in climate models. To illustrate the use of this benchmark, two very different atmospheric dynamical cores--one spectral, one finite difference--are compared. It is found that the long-term statistics produced by the two models are very similar. Selected results from these calculations are presented to initiate the intercomparison.

Held, Isaac M.↗

The Community Land Model Version 5: Description of New Features, Benchmarking, and Impact of Forcing Uncertainty

The Community Land Model (CLM) is the land component of the Community Earth System Model (CESM) and is used in several global and regional modeling systems. In this paper, we introduce model developments included in CLM version 5 (CLM5), which is the default land component for CESM2. We assess an ensemble of simulations, including prescribed and prognostic vegetation state, multiple forcing data sets, and CLM4, CLM4.5, and CLM5, against a range of metrics including from the International Land Model Benchmarking (ILAMBv2) package. CLM5 includes new and updated processes and parameterizations: (1) dynamic land units, (2) updated parameterizations and structure for hydrology and snow (spatially explicit soil depth, dry surface layer, revised groundwater scheme, revised canopy interception and canopy snow processes, updated fresh snow density, simple firn model, and Model for Scale Adaptive River Transport), (3) plant hydraulics and hydraulic redistribution, (4) revised nitrogen cycling (flexible leaf stoichiometry, leaf N optimization for photosynthesis, and carbon costs for plant nitrogen uptake), (5) global crop model with six crop types and time-evolving irrigated areas and fertilization rates, (6) updated urban building energy, (7) carbon isotopes, and (8) updated stomatal physiology. New optional features include demographically structured dynamic vegetation model (Functionally Assembled Terrestrial Ecosystem Simulator), ozone damage to plants, and fire trace gas emissions coupling to the atmosphere. Conclusive establishment of improvement or degradation of individual variables or metrics is challenged by forcing uncertainty, parametric uncertainty, and model structural complexity, but the multivariate metrics presented here suggest a general broad improvement from CLM4 to CLM5.

58 GEOSCIENCES↗