Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “model benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 253 records · Page 14
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

Nuclear Dependence of Charged Current Inclusive $\bar{\nu}_\mu -$A Cross Section Measurements at 6 GeV at MINERvA

The main advantage of the MINERvA detector is that it uses differently sized nuclear targets, in the same neutrino beam, allowing for a direct study of nuclear medium effects, with reduced flux and detector uncertainties. This thesis presents the first high statistics charged current antineutrino-nucleus inclusive two-dimensional cross section measurements using an antineutrino beam with $\langle E_{\bar\nu}\rangle\sim$ 6 GeV, on iron, lead, carbon and hydrocarbon targets, as a function of bjorken $x$ and four-momentum transfer squared $Q^2$, invariant mass $W$ and $Q^2$ and muon longitudinal and transverse momenta, $p_t$ and $p_z$. These nuclei of different sizes were utilized to measure the nuclear dependence of the cross section. The antineutrino interactions that were utilized for the analysis presented in this thesis, were having an antimuon in the final state with energy 2-20GeV and with an antimuon scattering angle less than $17^o$. This analysis represents one of the largest antineutrino datasets analyzed to date in the energy region of $\langle E_{\bar\nu} \rangle \sim$ 6 GeV. This direct evidence for the nuclear medium effects, can help benchmarking models, which can be used by future neutrino experiments like DUNE, by extrapolating these results to Argon target.

Gaur, Prameet Kumar↗
DOE OSTI2024

INL Poster - Juan Barrera Salazar

Generation IV nuclear reactors introduce several advantages and benefits in terms of safety and efficiency when compared with their predecessors from previous generation. This is due, among many things, to the use of innovative forms of fuel and coolant, different from the conventional ones used in the last decades. Given that these upcoming designs utilize emerging technologies, the related instrumentation is also in the process of being developed; therefore, it is necessary to establish the sensitivity requirements and the effects of uncertainty on different properties of the components and elements of the reactor designs. This report presents the results of simulations that quantify the impacts of the uncertainties of four thermophysical properties of the refrigerant salt (LiF-BeF2) for the Kairos Power benchmark model (g-FHR) for steady state making use of the Sobol’ method through polynomial chaos surrogate modeling. The properties of the salt to which uncertainty was evaluated were density, dynamic viscosity, thermal conductivity and heat capacity. This study was carried out using the Griffin/Pronghorn multiphysics model under the computational resources of the Idaho National Laboratory (INL) High Performance Computing (HPC). The results indicate a weak dependence of the uncertainty of thermal conductivity on the quantities of core pressure drop and core outlet temperature.

21 - SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLAN↗
NASA NTRS1993

A determination of the external forces required to move the benchmark active controls testing model in pure plunge and pure pitch

In view of the strong need for a well-documented set of experimental data which is suitable for the validation and/or calibration of modern Computational Fluid Dynamics codes, the Benchmark Models Program was initiated by the Structural Dynamics Division of the NASA Langley Research Center. One of the models in the program, the Benchmark Active Controls Testing Model, consists of a rigid wing of rectangular planform with a NACA 0012 profile and three control surfaces (a trailing-edge control surface, a lower-surface spoiler, and an upper-surface spoiler). The model is affixed to a flexible mount system which allows only plunging and/or pitching motion. An approximate analytical determination of the forces required to move this model, with its control surfaces fixed, in pure plunge and pure pitch at a number of test conditions is included. This provides a good indication of the type of actuator system required to generate the aerodynamic data resulting from pure plunging and pure pitching motion, in which much interest was expressed. The analysis makes use of previously obtained numerical results.

Dcruz, Jonathan↗
NASA NTRS2000

Passivity-based Robust Control of Aerospace Systems

This report provides a brief summary of the research work performed over the duration of the cooperative research agreement between NASA Langley Research Center and Kansas State University. The cooperative agreement which was originally for the duration the three years was extended by another year through no-cost extension in order to accomplish the goals of the project. The main objective of the research was to develop passivity-based robust control methodology for passive and non-passive aerospace systems. The focus of the first-year's research was limited to the investigation of passivity-based methods for the robust control of Linear Time-Invariant (LTI) single-input single-output (SISO), open-loop stable, minimum-phase non-passive systems. The second year's focus was mainly on extending the passivity-based methodology to a larger class of non-passive LTI systems which includes unstable and nonminimum phase SISO systems. For LTI non-passive systems, five different passification. methods were developed. The primary effort during the years three and four was on the development of passification methodology for MIMO systems, development of methods for checking robustness of passification, and developing synthesis techniques for passifying compensators. For passive LTI systems optimal synthesis procedure was also developed for the design of constant-gain positive real controllers. For nonlinear passive systems, numerical optimization-based technique was developed for the synthesis of constant as well as time-varying gain positive-real controllers. The passivity-based control design methodology developed during the duration of this project was demonstrated by its application to various benchmark examples. These example systems included longitudinal model of an F-18 High Alpha Research Vehicle (HARV) for pitch axis control, NASA's supersonic transport wind tunnel model, ACC benchmark model, 1-D acoustic duct model, piezo-actuated flexible link model, and NASA's Benchmark Active Controls Technology (BACT) Wing model. Some of the stability results for linear passive systems were also extended to nonlinear passive systems. Several publications and conference presentations resulted from this research.

Kelkar, Atul G.↗
DOE OSTI2026

The AWAKEN wind farm benchmark, Part 2: Modeling results

Accurately modeling wind farm performance in complex atmospheric flows remains a challenge. This paper presents the modeling results of the American WAKE experimeNt (AWAKEN) wind farm benchmark, a collaborative effort involving 16 research groups from academia and industry within the International Energy Agency Wind Technology Collaboration Programme Task 57. The study evaluates a diverse suite of simulation tools, ranging from fast-running engineering wake models to high-fidelity large-eddy simulations, against a diurnal case study observed during the AWAKEN campaign. The benchmark utilized a three-phase structure to progressively assess model performance as observational data availability increased. Initial blind predictions showed that higher-fidelity models did not uniformly outperform simpler simulation tools. A distinct spatial bias was observed where models struggled to resolve the interplay between a low-level jet, wakes, and terrain-induced flow acceleration. In subsequent phases, leveraging additional measurements for model improvement led to a reduction in mean absolute error across the model ensemble; however, this effect was most pronounced in engineering wake models, where targeted calibration reduced error by up to 40~\%. Overall, the study demonstrates that inflow characterization remains a primary prerequisite for accuracy, particularly for models relying on coarse forcing datasets. While the limited ability to resolve local terrain-flow interactions under single-day conditions represent a recognized constraint, the overall findings on wake modeling and real-world validation still provide valuable guidance for model application and for mitigating this limitation.

Bodini, Nicola↗
DOE OSTI2021

Highly Accurate Experimental Heave Decay Tests with a Floating Sphere: A Public Benchmark Dataset for Model Validation of Fluid–Structure Interaction

Highly accurate and precise heave decay tests on a sphere with a diameter of 300 mm were completed in a meticulously designed test setup in the wave basin in the Ocean and Coastal Engineering Laboratory at Aalborg University, Denmark. The tests were dedicated to providing a rigorous benchmark dataset for numerical model validation. The sphere was ballasted to half submergence, thereby floating with the waterline at the equator when at rest in calm water. Heave decay tests were conducted, wherein the sphere was held stationary and dropped from three drop heights: a small drop height, which can be considered a linear case, a moderately nonlinear case, and a highly nonlinear case with a drop height from a position where the whole sphere was initially above the water. The precision of the heave decay time series was calculated from random and systematic standard uncertainties. At a 95% confidence level, uncertainties were found to be very low—on average only about 0.3% of the respective drop heights. Physical parameters of the test setup and associated uncertainties were quantified. A test case was formulated that closely represents the physical tests, enabling the reader to do his/her own numerical tests. The paper includes a comparison of the physical test results to the results from several independent numerical models based on linear potential flow, fully nonlinear potential flow, and the Reynolds-averaged Navier–Stokes (RANS) equations. A high correlation between physical and numerical test results is shown. The physical test results are very suitable for numerical model validation and are public as a benchmark dataset.

42 ENGINEERING↗
DOE OSTI2025

Benchmarking large language models for materials synthesis: The case of atomic layer deposition

In this work, we introduce an open-ended question benchmark, ALDbench, to evaluate the performance of large language models (LLMs) in materials synthesis, and, in particular, in the field of atomic layer deposition, a thin film growth technique used in energy applications and microelectronics. Our benchmark comprises questions with a level of difficulty ranging from the graduate level to domain expert current with the state of the art in the field. Human experts reviewed the questions along the criteria of difficulty and specificity, and the model responses along four different criteria: overall quality, specificity, relevance, and accuracy. We ran this benchmark on an instance of OpenAI’s GPT-4o. The responses from the model received a composite quality score of 3.7 on a 1–5 scale, consistent with a passing grade. However, 36% of the questions received at least one below average score. An in-depth analysis of the responses identified at least five instances of suspected hallucination. Finally, we observed statistically significant correlations between the difficulty of the question and the quality of the response, the difficulty of the question and the relevance of the response, the specificity of the question, and the accuracy of the response as graded by the human experts. Furthermore, this emphasizes the need to evaluate LLMs across multiple criteria beyond difficulty or accuracy.

Artificial intelligence↗
DOE OSTI2020

Alaskan carbon-climate feedbacks will be weaker than inferred from short-term manipulations: Alaskan Benchmark Data and Model runs

This submission aimed to assess differences in short-term step warming manipulations and long-term chronic response to climate change in Alaskan ecosystems. Briefly, climate warming is occurring fastest at high latitudes. Based on short-term field experiments, this warming is projected to stimulate soil organic matter decomposition, and promote a positive feedback to climate change. We show here that the tightly coupled, nonlinear nature of high-latitude ecosystems implies that short-term (< 10 year) warming experiments produce emergent ecosystem carbon stock temperature sensitivities inconsistent with emergent multi-decadal responses. We first demonstrate that a well-tested mechanistic ecosystem model accurately represents observed carbon cycle and active layer depth responses to short-term summer warming in four diverse Alaskan sites. We then show that short-term warming manipulations do not capture the non-linear, long-term dynamics of vegetation, and thereby soil organic matter, that occur in response to thermal, hydrological, and nutrient transformations belowground. Our results demonstrate significant spatial heterogeneity in multi-decadal Arctic carbon cycle trajectories and argue for more mechanistic models to improve predictive capabilities.The model used in the current study is available publicly (https://github.com/jinyun1tang/ECOSYS), and the current submission contains the python/ matlab codes for analyzing output from the model (includng a readme file to explain the codes). The benchmark data, also enclosed, was collected from a range of published and publicly available sources (extracted using GRABIT: https://www.mathworks.com/matlabcentral/fileexchange/7173-grabit). These sources describe warming induced changes in tundra/ boreal ecosystems.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation

As machine learning (ML) is increasingly implemented in hardware to address real-time challenges in scientific applications, the development of advanced toolchains has significantly reduced the time required to iterate on various designs. These advancements have solved major obstacles, but also exposed new challenges. For example, processes that were not previously considered bottlenecks, such as hardware synthesis, are becoming limiting factors in the rapid iteration of designs. To mitigate these emerging constraints, multiple efforts have been undertaken to develop an ML-based surrogate model that estimates resource usage of ML accelerator architectures. We introduce wa-hls4ml, a benchmark for ML accelerator resource and latency estimation, and its corresponding initial dataset of over 680,000 fully connected and convolutional neural networks, all synthesized using hls4ml and targeting Xilinx FPGAs. The benchmark evaluates the performance of resource and latency predictors against several common ML model architectures, primarily originating from scientific domains, as exemplar models, and the average performance across a subset of the dataset. Additionally, we introduce GNN- and transformer-based surrogate models that predict latency and resources for ML accelerators. We present the architecture and performance of the models and find that the models generally predict latency and resources for the 75% percentile within several percent of the synthesized resources on the synthetic test dataset.

Hawks, Benjamin [Fermilab] (ORCID:0000000157000288↗
DOE OSTI2025

Benchmark of numerical modeling approaches on the systematic performance evaluation of wave energy converters

Different numerical modeling methods have been developed and applied to evaluate a variety of performance indicators of wave energy converters (WECs), including the power performance, structural loads, levelized cost of energy, etc. Based on the modeling fidelity, the commonly used numerical modeling approaches can be classified as linear modeling, weakly nonlinear modeling and fully nonlinear modeling approaches. Each method differs in accuracy and computational efficiency, making them suitable for different stages of WEC design. However, the selection of modeling approach could significantly impact evaluation outcomes. For instance, simplified linear models may underestimate structural loads or overestimate energy production in some operational conditions, potentially leading to less cost-effective designs. Given the widespread utilization of these models, it is essential to understand the uncertainties brought by them in performance evaluations. This work is dedicated to benchmarking different linear-potential-flow-based numerical models for evaluating the systematic performance of WECs. Three representative numerical modeling approaches are considered in this work, including linear frequency-domain modeling, statistically linearized spectral-domain modeling and Cummins equation-based nonlinear time-domain modeling. A generic point absorber WEC is considered as the research reference in this work, and different sea sites are taken into account. The numerical models are utilized to predict critical performance indicators, including power performance, the annual energy production, the capacity factor, the levelized cost of energy and the PTO fatigue loads. By comparing the results, this work identifies the uncertainties associated with different modeling approaches in evaluating WEC performance.

Fatigue↗
NASA NTRS2019

A Segmentation Algorithm for Characterizing Rise and Fall Segments in Seasonal Cycles: an Application to Xco2 to Estimate Benchmarks and Assess Model Bias

There is more useful information in the time series of satellite-derived column-averaged carbon dioxide (XCO2) than is typically characterized. Often, the entire time series is treated at once without considering detailed features at shorter timescales, such as nonstationary changes in signal characteristics – amplitude, period and phase. In many instances, signals are visually and analytically differentiable from other portions in a time series. Each rise (increasing) and fall (decreasing) segment in the seasonal cycle is visually discernable in a graph of the time series. The rise and fall segments largely result from seasonal differences in terrestrial ecosystem production, which means that the segment's signal characteristics can be used to establish observational benchmarks because the signal characteristics are driven by similar underlying processes. We developed an analytical segmentation algorithm to characterize the rise and fall segments in XCO2 seasonal cycles. We present the algorithm for general application of the segmentation analysis and emphasize here that the segmentation analysis is more generally applicable to cyclic time series. We demonstrate the utility of the algorithm with specific results related to the comparison between satellite- and model-derived XCO2 seasonal cycles (2009–2012) for large bioregions across the globe. We found a seasonal amplitude gradient of 0.74–0.77 ppm for every 10∘ of latitude in the satellite data, with similar gradients for rise and fall segments. This translates to a south–north seasonal amplitude gradient of 8 ppm for XCO2, about half the gradient in seasonal amplitude based on surface site in situ CO2 data (∼19 ppm). The latitudinal gradients in the period of the satellite-derived seasonal cycles were of opposing sign and magnitude (−9 d per 10∘ latitude for fall segments and 10 d per 10∘ latitude for rise segments) and suggest that a specific latitude (∼2∘ N) exists that defines an inversion point for the period asymmetry. Before (after) the point of asymmetry inversion, the periods of rise segments are lesser (greater) than the periods of fall segments; only a single model could reproduce this emergent pattern. The asymmetry in amplitude and the period between rise and fall segments introduces a novel pattern in seasonal cycle analyses, but, while we show these emergent patterns exist in the data, we are still breaking ground in applying the information for science applications. Maybe the most useful application is that the segmentation analysis allowed us to decompose the model biases into their correlated parts of biases in amplitude, period and phase independently for rise and fall segments. We offer an extended discussion on how such information about model biases and the emergent patterns in satellite-derived seasonal cycles can be used to guide future inquiry and model development.

segmentation algorithm↗
DOE OSTI2026

A curated benchmark for cofolding models on kinase conformational states

Abstract Protein kinases are critical drug targets, requiring therapeutics that can modulate their active and inactive conformational states. While cofolding models can generate global folds directly from kinase sequences and ligand SMILES strings, these models have not yet been tested on their ability to recover ligand-induced-fit conformational states of the kinase proteins. Here, we introduce KinConfBench, a curated benchmark of 2225 high-quality human kinase chains to evaluate the ability of four state-of-the-art cofolding models—Boltz-2, Chai-1, Protenix, and RoseTTAFold-All-Atom—to recover both canonical and rare conformational states. We show that geometric success metrics of a ligand pose in the active site do not correlate strongly with the correct kinase conformational state, motivating a new set of dynamical benchmarks for assessing cofolding models. While all four cofolding models achieve ~60–80% prediction accuracy for kinase conformational classification, they exhibit severe mode collapse when performing multiple inferences, show negligible structural diversity in sampling induced-fit motions, and display a prevalent “apo-drift” in which most cofolding models predominantly predict the kinase to be in its ligand-free state. Our results highlight that capturing ligand-induced protein conformational diversity, not just geometric fit, is critical for next-generation structure-based drug discovery.

Sun, Kunyang↗
DOE OSTI2025

wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation

As machine learning (ML) is increasingly implemented in hardware to address real-time challenges in scientific applications, the development of advanced toolchains has significantly reduced the time required to iterate on various designs. These advancements have solved major obstacles, but also exposed new challenges. For example, processes that were not previously considered bottlenecks, such as hardware synthesis, are becoming limiting factors in the rapid iteration of designs. To mitigate these emerging constraints, multipleefforts have been undertaken to develop an ML-based surrogate model that estimates resource usage of synthesized ML accelerator architectures. We introduce wa-hls4ml, a benchmark for ML accelerator resource and latency estimation, and its corresponding initial dataset of over 680 000 fully connected and convolutional neural networks, all synthesized using hls4ml and targeting Xilinx FPGAs. The benchmark evaluates the performance of resource and latency predictors against several common ML model architectures, primarily originating from scientific domains, as exemplar models, and the average performance across a subset of the dataset. Additionally, we introduce GNN- and transformer-based surrogate models that predict latency and resources for ML accelerators. We present the architecture and performance of the models and find that the models generally predict latency and resources for the 75% percentile within several percent of the synthesized resources on the synthetic test dataset.

Hawks, Benjamin G. [Fermilab]↗
NASA NTRS1993

Four-Nozzle Benchmark Wind Tunnel Model USA Code Solutions for Simulation of Multiple Rocket Base Flow Recirculation at 145,000 Feet Altitude

Multiple rocket exhaust plume interactions at high altitudes can produce base flow recirculation with attendant alteration of the base pressure coefficient and increased base heating. A search for a good wind tunnel benchmark problem to check grid clustering technique and turbulence modeling turned up the experiment done at AEDC in 1961 by Goethert and Matz on a 4.25-in. diameter domed missile base model with four rocket nozzles. This wind tunnel model with varied external bleed air flow for the base flow wake produced measured p/p(sub ref) at the center of the base as high as 3.3 due to plume flow recirculation back onto the base. At that time in 1961, relatively inexpensive experimentation with air at gamma = 1.4 and nozzle A(sub e)/A of 10.6 and theta(sub n) = 7.55 deg with P(sub c) = 155 psia simulated a LO2/LH2 rocket exhaust plume with gamma = 1.20, A(sub e)/A of 78 and P(sub c) about 1,000 psia. An array of base pressure taps on the aft dome gave a clear measurement of the plume recirculation effects at p(infinity) = 4.76 psfa corresponding to 145,000 ft altitude. Our CFD computations of the flow field with direct comparison of computed-versus-measured base pressure distribution (across the dome) provide detailed information on velocities and particle traces as well eddy viscosity in the base and nozzle region. The solution was obtained using a six-zone mesh with 284,000 grid points for one quadrant taking advantage of symmetry. Results are compared using a zero-equation algebraic and a one-equation pointwise R(sub t) turbulence model (work in progress). Good agreement with the experimental pressure data was obtained with both; and this benchmark showed the importance of: (1) proper grid clustering and (2) proper choice of turbulence modeling for rocket plume problems/recirculation at high altitude.

Dougherty, N. S.↗
DOE OSTI2026

Cooling Matters: Benchmarking Large Language Models and Vision-Language Models on Liquid-Cooled Versus Air-Cooled H100 GPU Systems

The unprecedented growth in artificial intelligence (AI) workloads, recently dominated by large language models (LLMs) and vision-language models (VLMs), has intensified power and cooling demands in data centers. This study benchmarks LLMs and VLMs on two HGX nodes, each with 8× NVIDIA H100 graphics processing units (GPUs), using liquid and air cooling. Leveraging GPU Burn, Weights & Biases, and IPMItool, we collect detailed thermal, power, and computation data. Results show that the liquid-cooled systems maintain GPU temperatures between 41-50$^\circ$C, while the air-cooled counterparts fluctuate between 54-72$^\circ$C under load. This thermal stability of liquid-cooled systems yields 17% higher performance (54 TFLOPs/ GPU vs. 46 TFLOPs/GPU), performance-per-watt, reduced energy overhead, and greater system efficiency than the air-cooled counterparts. These findings underscore the energy and sustainability benefits of liquid cooling, offering a compelling path forward for hyperscale data centers seeking to optimize AI infrastructure. https://github.com/iscaas/Cooling-Matters.

Latif, Imran↗
DOE OSTI2022

The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider

We describe the outcome of a data challenge conducted as part of the Dark Machines (https://www.darkmachines.org) initiative and the Les Houches 2019 workshop on Physics at TeV colliders. The challenged aims to detect signals of new physics at the Large Hadron Collider (LHC) using unsupervised machine learning algorithms. First, we propose how an anomaly score could be implemented to define model-independent signal regions in LHC searches. We define and describe a large benchmark dataset, consisting of >1 billion simulated LHC events corresponding to 10\, fb^{-1} 10 f b − 1 of proton-proton collisions at a center-of-mass energy of 13 TeV. We then review a wide range of anomaly detection and density estimation algorithms, developed in the context of the data challenge, and we measure their performance in a set of realistic analysis environments. We draw a number of useful conclusions that will aid the development of unsupervised new physics searches during the third run of the LHC, and provide our benchmark dataset for future studies at https://www.phenoMLdata.org. Code to reproduce the analysis is provided at https://github.com/bostdiek/DarkMachines-UnsupervisedChallenge.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2020

Evaluating three evapotranspiration estimates from model of different complexity over China using the ILAMB benchmarking system

The land surface models range in complexity of terrestrial evapotranspiration, yet it is unknown how model complexity translates to accuracy of modeled evapotranspiration estimates. Here, we use the International Land Model Benchmarking system to assess ET estimates from three models of varying complexity driven by the same forcing datasets: an earth system model, a terrestrial biosphere model, and a stand-alone ET model. The performance assessment includes both temporal and spatial evaluation, and different plant functional types across China. Our results indicate that the most complex model, an earth system model, performed best against the benchmarking datasets and metrics. Terrestrial biosphere model performed best in simulating inter-annual variability of ET, while earth system model performed best in simulating the seasonal cycle. The more complex models (earth system model and terrestrial biosphere model) perform better in forest, shrub and crop ecosystems, while the simpler model (stand-alone ET model) perform better in grass ecosystems. Our study demonstrates the impact of model complexity on ET estimates and highlights directions for future ET model improvements.

54 ENVIRONMENTAL SCIENCES↗