SEARCH · Engineering Papers

Results for “model benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 739 records · Page 41

All sources NASA NTRS DOE OSTI

DOE OSTI2026

Multi‐Model Ensembles in Ecosystem Modeling: Challenges and Best Practices for Decision‐Making

Ecosystem models are increasingly central to the decision-making for environmental policy, conservation planning, and climate-related investments. Yet, the growing reliance on Multi-Model Ensembles (MMEs) of ecosystem models by practitioners and policymakers, sometimes under tight timelines and imperfect information, has frequently outpaced the scientific rigor required to ensure ensemble reliability. Here, MMEs refer to approaches that combine targeted predictions from multiple models with the expectation of improving robustness and quantifying predictive uncertainty. Poorly designed MMEs may create a false sense of confidence and lead to suboptimal policy and market decisions. This perspective argues that robust decision-making-relevant MMEs must be grounded on two pillars: (1) rigorous Model Intercomparison Projects (MIPs), which identify inter-model agreement and disagreement, characterize model uncertainties, and evaluate robustness with observationally based benchmarks—MIPs' diagnostic evaluation is so critical that it must be needed to drive MME's decision in model selection and weighting, especially when only a limited number of models available; and (2) co-design by both stakeholders and scientists to ensure that scenarios, metrics and uncertainty requirements provide decision-relevant information. Building upon the past success and lessons from the existing MIPs-MMEs efforts (e.g., climate/Earth system/crop), we derived the theoretical basis for MMEs, addressed their specific challenges in ecosystem modeling, and highlighted proper consideration of model numbers and diversity, risk of model inter-dependence, effective calibration of model parameters, possible overdue of some ecosystem model development, critical roles of open benchmark data across a wide range of conditions, and suggested use of Artificial Intelligence to support MIPs-MMEs. We highlighted the under-recognized opportunity for MIPs and MMEs to drive scientific progress and innovation through identifying better performing models, systematic benchmarking, feedback loops, and targeted model improvement. By following actionable best practice guidelines, MMEs can evolve from ad hoc aggregation of models into a trusted backbone of environmental policy and decision-making.

ecosystem modeling↗

DOE OSTI2023

Model metamers reveal divergent invariances between biological and artificial neural networks

Deep neural network models of sensory systems are often proposed to learn representational transformations with invariances like those in the brain. To reveal these invariances, we generated ‘model metamers’, stimuli whose activations within a model stage are matched to those of a natural stimulus. Metamers for state-of-the-art supervised and unsupervised neural network models of vision and audition were often completely unrecognizable to humans when generated from late model stages, suggesting differences between model and human invariances. Targeted model changes improved human recognizability of model metamers but did not eliminate the overall human–model discrepancy. The human recognizability of a model’s metamers was well predicted by their recognizability by other models, suggesting that models contain idiosyncratic invariances in addition to those required by the task. Metamer recognizability dissociated from both traditional brain-based benchmarks and adversarial vulnerability, revealing a distinct failure mode of existing sensory models and providing a complementary benchmark for model assessment.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2023

Quantum Simulation of an Open System: A Dissipative 1+1D Ising Model

The 1+1D Ising model is an ideal benchmark for quantum algorithms, as it is very well understood theoretically. This is true even when expanding the model to include complex coupling constants. In this work, we implement quantum algorithms designed for the simulation of open or complex coupling quantum field theories on IBM devices with a focus on the measurement of the Lee-Yang edge singularity. This feature corresponds (at large volumes) to a phase transition, and our successful reproduction of the transition represents a non-trivial test for current hardware and its ability to distinguish features of interest in quantum field theories.

Gustafson, Erik↗

DOE OSTI2025

tether

Tether is a python module for benchmarking and assessing large language model (LLMs) performance at generic scientific tasks. The code generates benchmarks, uses the benchmark to prompt LLMs through automatic programming interfaces (APIs), and then logs the number of prompts an LLM correctly answers and presents the results as a completed benchmark.

Kaiser, Bryan [Los Alamos National Laboratory]↗

DOE OSTI2020

Results of the third Marine Ice Sheet Model Intercomparison Project (MISMIP+)

We present the result of the third Marine Ice Sheet Model Intercomparison Project, MISMIP+. MISMIP+ is intended to be a benchmark for ice-flow models which include fast sliding marine ice streams and floating ice shelves and in particular a treatment of viscous stress that is sufficient to model buttressing, where upstream ice flow is restrained by a downstream ice shelf. A set of idealized experiments first tests that models are able to maintain a steady state with the grounding line located on a retrograde slope due to buttressing and then explore scenarios where a reduction in that buttressing causes ice stream acceleration, thinning, and grounding line retreat. The majority of participating models passed the first test and then produced similar responses to the loss of buttressing. We find that the most important distinction between models in this particular type of simulation is in the treatment of sliding at the bed, with other distinctions – notably the difference between the simpler and more complete treatments of englacial stress but also the differences between numerical methods – taking a secondary role.

58 GEOSCIENCES↗

NASA NTRS2004

Effects of Angle of Attack and Velocity on Trailing Edge Noise

Trailing edge (TE) noise measurements for a NACA 63-215 airfoil model are presented, providing benchmark experimental data for a cambered airfoil. The effects of flow Mach number and angle of attack of the airfoil model with different TE bluntnesses are shown. Far-field noise spectra and directivity are obtained using a directional microphone array. Standard and diagonal removal beamforming techniques are evaluated employing tailored weighting functions for quantitatively accounting for the distributed line character of TE noise. Diagonal removal processing is used for the primary database as it successfully removes noise contaminates. Some TE noise predictions are reported to help interpret the data, with respect to flow speed, angle of attack, and TE bluntness on spectral shape and peak levels. Important findings include the validation of a TE noise directivity function for different airfoil angles of attack and the demonstration of the importance of the directivity function s convective amplification terms.

Hutcheson, Florence V.↗

NASA NTRS2006

Effects of Angle of Attack and Velocity on Trailing Edge Noise

Hutcheson, Florence V.↗

DOE OSTI2024

SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions

Instruction finetuning is a popular paradigm to align large language models (LLM) with human intent. Despite its popularity, this idea is less explored in improving the LLMs to align existing foundation models with scientific disciplines, concepts and goals. In this work, we present SciTune as a tuning framework to improve the ability of LLMs to follow scientific multimodal instructions. To test our methodology, we use a human-generated scientific instruction tuning dataset and train a large multimodal model LLaMA-SciTune that connects a vision encoder and LLM for science-focused visual and language understanding. LLaMA-SciTune significantly outperforms the state-of-the-art models in the generated figure types and captions in multiple scientific multimodal benchmarks. In comparison to the models that are fine-tuned with machine generated data only, LLaMA-SciTune surpasses human performance on average and in many sub-categories on the ScienceQA benchmark.

• Artificial intelligence (AI) / machine learning ↗

DOE OSTI2023

Dynamics of streamflow permanence in a headwater network: Insights from catchment-scale model simulations

The hillslope and channel dynamics that govern streamflow permanence in headwater systems have important implications for ecosystem functioning and downstream water quality. Recent advancements in process-based, semi-distributed hydrologic models that build upon empirical studies of streamflow permanence in well-monitored headwater catchments show promise for characterizing the dynamics of streamflow permanence in headwater systems. However, few process-based models consider the continuum of hillslope-stream network connectivity as a control on streamflow permanence in headwater systems. The objective of this study was to expand a process-based, catchment-scale hydrologic model to better understand the spatiotemporal dynamics of headwater streamflow permanence and to identify controls of streamflow expansion and contraction in a headwater network. Further, we aimed to develop an approach that enhanced the fidelity of model simulations, yet required little additional data, with the intent that the model might be later transferred to catchments with limited long-term and spatially explicit measurements. This approach facilitated network-scale estimates of the controls of streamflow expansion and contraction, albeit with higher degrees of uncertainty in individual reaches due to data constraints. Our model simulated that streamflow permanence was highly dynamic in first-order reaches with steep slopes and variable contributing areas. The simulated stream network length ranged from nearly 98±2% of the geomorphic channel extent during wet periods to nearly 50±10% during dry periods. The model identified a discharge threshold of approximately 1 mm d –1 , above which the rate of streamflow expansion decreases by nearly an order of magnitude, indicating a lack of sensitivity of streamflow expansion to hydrologic forcing during high-flow periods. Overall, we demonstrate that process-based, catchment-scale models offer important insights on the controls of streamflow permanence, despite uncertainties and limitations of the model. We encourage researchers to increase data collection efforts and develop benchmarks to better evaluate such models.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2023

Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction

Abstract Background West Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental USA. WNV occurrence has high spatiotemporal variation, and current approaches to targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement. Methods We performed a multi-model comparative assessment of probabilistic forecasts submitted by 15 teams for annual WNND cases in US counties for 2020 and assessed forecast accuracy, calibration, and discriminatory power. In the evaluation, we included forecasts produced by comparison models of varying complexity as benchmarks of forecast performance. We also used regression analysis to identify modeling approaches and contextual factors that were associated with forecast skill. Results Simple models based on historical WNND cases generally scored better than more complex models and combined higher discriminatory power with better calibration of uncertainty. Forecast skill improved across updated forecast submissions submitted during the 2020 season. Among models using additional data, inclusion of climate or human demographic data was associated with higher skill, while inclusion of mosquito or land use data was associated with lower skill. We also identified population size, extreme minimum winter temperature, and interannual variation in WNND cases as county-level characteristics associated with variation in forecast skill. Conclusions Historical WNND cases were strong predictors of future cases with minimal increase in skill achieved by models that included other factors. Although opportunities might exist to specifically improve predictions for areas with large populations and low or high winter temperatures, areas with high case-count variability are intrinsically more difficult to predict. Also, the prediction of outbreaks, which are outliers relative to typical case numbers, remains difficult. Further improvements to prediction could be obtained with improved calibration of forecast uncertainty and access to real-time data streams (e.g. current weather and preliminary human cases). Graphical Abstract

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2020

A Metallic Fuel Performance Benchmark Problem Based on the IFR-1 Experiment

Metallic nuclear fuels are an active area of research and development for use in advanced reactors. Robust and accurate metallic fuel performance models are necessary for the design, safety analysis, and licensing of such reactors. However, metallic fuel performance models are not as mature as ceramic fuel performance models and require additional development. In order to better gauge the accuracy of existing models, identify high-priority models for development, and quantify the improvements made by future model development, ORNL has developed a benchmark problem based on the IFR-1 experiment. This work gathers all relevant information on the IFR-1 experiment and uses it to develop the benchmark problem. It simulates the problem using the fuel performance code BISON and compare the results to post-irradiation examination data from the experiment. Then a sensitivity analysis is performed on the BISON model to determine the benchmark problem’s sensitivity to uncertainty in the input parameters. The results suggest that BISON’s mechanical models require additional development. Plastic deformation is under-predicted in the cladding and axial swelling is over-predicted in the fuel. These problems may be related. Additionally, there is a bias in the temperature which may be a result of uncertainty in the input parameters rather than an issue with the fuel performance models.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

DOE OSTI2023

Improving E3SM Land Model Photosynthesis Parameterization via Satellite SIF, Machine Learning, and Surrogate Modeling

The parameterization of key photosynthesis parameters is one of the key uncertain sources in modeling ecosystem gross primary productivity (GPP). Solar-induced chlorophyll fluorescence (SIF) offers a good proxy for GPP since it marks the actual process of photosynthesis; while machine learning (ML) provides a robust approach to model the GPP-SIF relationship. Here, we trained the boosted regressing tree (BRT) and the Random Forest ML models with Greenhouse Gases Observing Satellite SIF data and in situ GPP observations from 49 eddy covariance towers. These trained ML GPP-SIF models were fed into the Energy Exascale Earth System Model (E3SM) Land Model (ELM) to generate ELM-simulated global SIF estimates, which were then benchmarked against satellite SIF observations with a surrogate modeling approach. Our results indicated good modeling performance of the ML-based GPP-SIF relationship. The ELM model when fed with the ML GPP-SIF models also can well predict the spatial-temporal variations in SIF. We also found high model accuracy for the surrogate modeling. Model parameter sensitivity analysis suggested that the fraction of leaf nitrogen in RuBisCO (flnr) is the most sensitive parameter to the SIF; other sensitive parameters include the Ball-Berry stomatal conductance slope (mbbopt) and the vcmax entropy (vcmaxse). The posterior uncertainty in simulated GPP was greatly reduced after benchmarking, and the model produced improved spatial patterns of mean GPP relative to FLUXCOM GPP. Our integrated approach provides a new avenue for improving land models and using remote-sensing SIF, which can be further improved in the future with more ground- and satellite-based observations.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS1985

Aerothermal modeling program, phase 2

The main objectives of the Aerothermal Modeling Program, Phase 2 are: to develop an improved numerical scheme for incorporation in a 3-D combustor flow model; to conduct a benchmark quality experiment to study the interaction of a primary jet with a confined swirling crossflow and to assess current and advanced turbulence and scalar transport models; and to conduct experimental evaluation of the air swirler interaction with fuel injectors, assessments of current two-phase models, and verification the improved spray evaporation/dispersion models.

Mongia, H. C.↗

DOE OSTI2023

Development of a Benchmark Eddy Flux Evapotranspiration Dataset for Evaluation of Satellite-Driven Evapotranspiration Models Over the CONUS

A large sample of ground-based evapotranspiration (ET) measurements made in the United States, primarily from eddy covariance systems, were post-processed to produce a benchmark ET dataset. The dataset was produced primarily to support the intercomparison and evaluation of the OpenET satellite-based remote sensing ET (RSET) models and could also be used to evaluate ET data from other models and approaches. OpenET is a web-based service that makes field-delineated and pixel-level ET estimates from well-established RSET models readily available to water managers, agricultural producers, and the public. The benchmark dataset is composed of flux and meteorological data from a variety of providers covering native vegetation and agricultural settings. Flux footprint predictions were developed for each station and included static flux footprints developed based on average wind direction and speed, as well as dynamic hourly footprints that were generated with a physically based model of upwind source area. The two footprint prediction methods were rigorously compared to evaluate their relative spatial coverage. Data from all sources were post-processed in a consistent and reproducible manner including data handling, gap-filling, temporal aggregation, and energy balance closure correction. The resulting dataset included 243,048 daily and 5,284 monthly ET values from 194 stations, with all data falling between 1995 and 2021. We assessed average daily energy imbalance using 172 flux sites with a total of 193,021 days of data, finding that overall turbulent fluxes were understated by about 12% on average relative to available energy. Multiple linear regression analyses indicated that daily average latent energy flux may be typically understated slightly more than sensible heat flux. This dataset was developed to provide a consistent reference to support evaluation of RSET data being developed for a wide range of applications related to water accounting and water resources management at field to watershed scales.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

Characterizing the multisectoral impacts of future global hydrologic variability

There is significant uncertainty in how global water supply will evolve in the future, due to uncertain climate, socioeconomic, and land use change drivers and variability of hydrologic processes. It is critical to characterize the potential impacts of uncertainty in future water supply given its importance for food and energy production. In this work, we introduce a framework that integrates stochastic hydrology and human-environmental systems to characterize uncertainty in future water supply and its multisector impacts. We develop a global stochastic watershed model and demonstrate that this model can generate a large ensemble of realizations of basin-scale runoff with global coverage that preserves the mean, variance, and spatial correlation of a historical benchmark. We couple this model with a well-known human-environmental systems model to explore the impacts of runoff variability on the water and agricultural sectors across spatial scales. We find that the impacts of future hydrologic variability vary across sectors and regions. Impacts are felt most strongly in the water and agricultural sectors for basins that are expected to have unsustainable water use in the future, such as the Indus River basin. For this basin, we find that the variability in future irrigation water withdrawals and irrigated cropland increase over time due to uncertainty in renewable water supply. We also use the Indus basin to show how our stochastic ensemble can be leveraged to explore the global multisector consequences of local extreme runoff conditions. This work introduces a novel technique to explore the propagation of future hydrologic variability across human and natural systems and spatial scales.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2023

Estimation of Participation Factors Using the Synchrosqueezed Wavelet Transform

This paper proposes a data-driven approach for estimating participation factors for a power system using only simulation results on selected disturbances. The approach is purely response-based and does not need a linearized system model for eigen-analysis, which makes it applicable to systems whose detailed, complete mathematical models are not available. Considering the unavoidable nonlinearity as exhibited in the transient period of a system response, the Synchrosqueezed Wavelet Transform is applied to simulated responses for modal analysis to obtain participation factors. Based on simulations of Kundur's two-area system using both the electromagnetic transient model and phasor model, the participation factors estimated by the proposed approach are compared with two other signal processing tools, the Prony analysis and continuous wavelet transform, and are also benchmarked with conventional model-based participation factors.

participation factor↗

DOE OSTI2021

U.S. Solar Photovoltaic BESS System Cost Benchmark Q1 2020 Report

NREL has been modeling U.S. solar photovoltaic (PV) system costs since 2009. This year, our report benchmarks costs of U.S. PV for residential, commercial, and utility-scale systems, with and without storage, built in the first quarter of 2020 (Q1 2020). The methodology includes bottom-up accounting for all system and project-development costs incurred when installing residential, commercial, and utility-scale systems and it models the capital costs and levelized cost of energy for such systems.

14 SOLAR ENERGY↗

DOE OSTI2025

A hybrid calorimetry-simulation model of mixing enthalpy for molten salt

Calorimetric determination of enthalpies of mixing (ΔH mix ) in multicomponent molten salts is often interpreted using empirical models that lack physically meaningful parameters. However, for improving pyrochemical separation of spent nuclear fuel, where lanthanides are major fission products and critical elements, a deeper thermodynamic understanding of the link between excess thermodynamic properties and solvation structure is critically needed. In this work, we implement a hybrid and physics-informed framework, MIVM+Calorimetry+AIMD, which integrates experimentally measured ΔH mix (via high temperature drop calorimetry) with solvation structures from ab initio molecular dynamics (AIMD). This approach is demonstrated using LaCl 3 mixed with eutectic LiCl-KCl (58 mol% – 42 mol%) at 873 K and 1133 K. MIVM-derived parameters enable extrapolation of excess Gibbs energy and La 3+ activity across compositions. In contrast, direct ΔH mix predictions from AIMD and polarizable ion model simulations deviate significantly. By incorporating experimentally benchmarked solvation structures into an interpretable thermodynamic model, the MIVM+Calorimetry+AIMD formalism achieves higher accuracy and generalizable method for studying molten salts, offering a robust path for understanding and optimizing molten salt chemistry relevant to nuclear fuel cycles and separation science.

Goncharov, Vitaliy G. [Washington State Univ., Pul↗