Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “BENCHMARKS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 937 records · Page 52
All sourcesNASA NTRSDOE OSTI
DOE OSTI2026

QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score.

Cao, Shuxiang↗
DOE OSTI2023

SABATH: Surrogate AI Benchmarking Applications' Testing Harness

SABATH provides benchmarking infrastructure for evaluating scientific ML/AI models. It contains support for scientific machine learning surrogates from external repositories. The software dependences are explicitly exposed in the surrogate model definition, which allows the use of advanced optimization, communication, and hardware features.

Luszczek, Piotr [Univ. of Tennessee, Knoxville, TN↗
DOE OSTI2023

Advances in benchmarking and round robin testing for PEM water electrolysis: Reference protocol and hardware

While the number of publications in the PEM water electrolysis community increases each year, no common ground concerning reference hardware (test cells and test bench) and testing protocols has been yet established. This would, however, be necessary for the comparability of experimental results. First attempts for such reference hardware and procedures have been made in the framework of the Task 30 Electrolysis within the Technology Collaboration Programme on Advanced Fuel Cells (AFC TCP) of the International Energy Agency (IEA). Since then, improvements of both the test hardware (test cell and components) as well as the measurement protocol were identified, and a revised methodology and key results based on a comprehensive measurement series have been obtained. A detailed protocol for testing commercial reference components with a reference laboratory test cell developed in-house by Fraunhofer ISE is presented. For evaluation of the protocol and the hardware, it was tested at three different institutions at the same time. Impedance spectroscopic and polarization data was acquired and analyzed. The obtained differences in performance were calculated to give the community an expectation window to compare own data to. Finally, the importance of a thorough temperature control and the conditioning phase are demonstrated.

08 HYDROGEN↗
DOE OSTI2020

Lattice physics calculations using the embedded self-shielding method in polaris, Part II: Benchmark assessment

Polaris is a 2D multigroup lattice physics capability in the SCALE code system used for the analysis of light water reactor fuel designs. In this work, verification and validation results of Polaris in SCALE 6.2 are presented. The SCALE/KENO continuous-energy Monte Carlo code is used as a reference to compare the results of more than 2,000 combinations of pressurized water reactor and boiling water reactor lattice designs and state points. Radiochemical assay measurements were also used to assess the accuracy of depletion calculations. Results show that Polaris, with ENDF/B-VII.1 cross section library, produces accurate lattice physics data for core simulators.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

Recent Advances in Immobilizing and Benchmarking Molecular Catalysts for Artificial Photosynthesis

Transition metal complexes have been widely used as catalysts or chromophores in artificial photosynthesis. Traditionally, they are employed in homogeneous settings. Despite their functional versatility and structural tunability, broad industrial applications of these catalysts are impeded by the limitations of homogeneous catalysis such as poor catalyst recyclability, solvent constraints (mostly organic solvents), and catalyst durability. Over the past few decades, researchers have developed various methods for molecular catalyst heterogenization to overcome these limitations. Here, in this review, we summarize recent developments in heterogenization strategies, with a focus on describing methods employed in the heterogenization process and their effects on catalytic performances. Alongside the in-depth discussion of heterogenization strategies, this review aims to provide a concise overview of the key metrics associated with heterogenized systems. We hope this review will aid researchers who are new to this research field in gaining a better understanding.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

A Practical Probabilistic Benchmark for AI Weather Models

Since the weather is chaotic, it is necessary to forecast an ensemble of future states. Recently, multiple AI weather models have emerged claiming breakthroughs in deterministic skill. Unfortunately, it is hard to fairly compare ensembles of AI forecasts because variations in ensembling methodology become confounding and the baseline data volume is immense. We address this by scoring lagged initial condition ensembles—whereby an ensemble can be constructed from a library of deterministic hindcasts. This allows the first parameter‐free intercomparison of leading AI weather models' probabilistic skill against an operational baseline. Lagged ensembles of the two leading AI weather models, GraphCast and Pangu, perform similarly even though the former outperforms the latter in deterministic scoring. These results are elaborated upon by sensitivity tests showing that commonly used multiple time‐step loss functions damage ensemble calibration.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Electronic structure analysis and DFT benchmarking of Rydberg-type alkali-metal-crown ether, -cryptand, and -adamanzane complexes

Density functional theory (DFT) and electron propagator theory (EPT) calculations were performed to study ground and excited electronic structures of alkali-metal (M) coordinated 9-crown-3, 24-crown-8, [2.1.1]cryptand, o-Me 2 -1.1.1, and 3 6 Adamanzane complexes. Each complex bears an expanded electron in the periphery and occupies diffuse 1p-, 1d-, 1f-type molecular orbitals (or superatomic 1P, 1D, 1F orbitals) in excited electronic states. The calculated superatomic shell model of the M(9-crown-3) 2 is 1S, 1P, 1D, 1F, 2S, 2P, 2D, 1G and it is held by all other complexes up to the studied 1F level. Due to the highly diffuse nature of the electron, the ionization energies of these complexes are significantly lower (1.6–2.0 eV) and hence these complexes belong to the superalkali category. Further, the ab initio EPT ionization energy and the excitation energies of the Li(9-crown-3) 2 were used to evaluate DFT errors associated with a series of exchange correlation functionals that span multiple rungs of Jacob's ladder (i.e., GGA, meta-GGA, global GGA hybrid, meta-GGA hybrid, range-separated hybrid, double-hybrid). Among these, the best performing functional is the range-separated hybrid CAM-B3LYP and the errors are within 6% of high-level ab initio EPT results. The accuracy of CAM-B3LYP is indeed transferable to similar complexes and hence the findings are expected to accelerate the progression of studies of Rydberg-type systems.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗