Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “benchmarks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

Critical Assessment of Metagenome Interpretation—a benchmark of metagenomics software

Methods for assembly, taxonomic profiling and binning are key to interpreting metagenome data, but a lack of consensus about benchmarking complicates performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on highly complex and realistic data sets, generated from ~700 newly sequenced microorganisms and ~600 novel viruses and plasmids and representing common experimental setups. Assembly and genome binning programs performed well for species represented by individual genomes but were substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below family level. Parameter settings markedly affected performance, underscoring their importance for program reproducibility. The CAMI results highlight current challenges but also provide a roadmap for software selection to answer specific research questions.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Benchmarking quantum logic operations relative to thresholds for fault tolerance

Contemporary methods for benchmarking noisy quantum processors typically measure average error rates or process infidelities. However, thresholds for fault-tolerant quantum error correction are given in terms of worst-case error rates—defined via the diamond norm—which can differ from average error rates by orders of magnitude. One method for resolving this discrepancy is to randomize the physical implementation of quantum gates, using techniques like randomized compiling (RC). In this work, we use gate set tomography to perform precision characterization of a set of two-qubit logic gates to study RC on a superconducting quantum processor. We find that, under RC, gate errors are accurately described by a stochastic Pauli noise model without coherent errors, and that spatially correlated coherent errors and non-Markovian errors are strongly suppressed. We further show that the average and worst-case error rates are equal for randomly compiled gates, and measure a maximum worst-case error of 0.0197(3) for our gate set. Our results show that randomized benchmarks are a viable route to both verifying that a quantum processor’s error rates are below a fault-tolerance threshold, and to bounding the failure rates of near-term algorithms, if—and only if—gates are implemented via randomization methods which tailor noise.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Quantum annealing for combinatorial optimization: a benchmarking study

Quantum annealing (QA) has the potential to significantly improve solution quality and reduce time complexity in solving combinatorial optimization problems compared to classical optimization methods. However, due to the limited number of qubits and their connectivity, the QA hardware did not show such an advantage over classical methods in past benchmarking studies. Recent advancements in QA with more than 5000 qubits, enhanced qubit connectivity, and the hybrid architecture promise to realize the quantum advantage. Here, we use a quantum annealer with state-of-the-art techniques and benchmark its performance against classical solvers. To compare their performance, we solve over 50 optimization problem instances represented by large and dense Hamiltonian matrices using quantum and classical solvers. The results demonstrate that a state-of-the-art quantum solver has higher accuracy (~0.013%) and a significantly faster problem-solving time (~6561×) than the best classical solver. Our results highlight the advantages of leveraging QA over classical counterparts, particularly in hybrid configurations, for achieving high accuracy and substantially reduced problem solving time in large-scale real-world optimization problems.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Benchmarking highly entangled states on a 60-atom analogue quantum simulator

Abstract Quantum systems have entered a competitive regime in which classical computers must make approximations to represent highly entangled quantum states 1,2 . However, in this beyond-classically-exact regime, fidelity comparisons between quantum and classical systems have so far been limited to digital quantum devices 2–5 , and it remains unsolved how to estimate the actual entanglement content of experiments 6 . Here, we perform fidelity benchmarking and mixed-state entanglement estimation with a 60-atom analogue Rydberg quantum simulator, reaching a high-entanglement entropy regime in which exact classical simulation becomes impractical. Our benchmarking protocol involves extrapolation from comparisons against an approximate classical algorithm, introduced here, with varying entanglement limits. We then develop and demonstrate an estimator of the experimental mixed-state entanglement 6 , finding our experiment is competitive with state-of-the-art digital quantum devices performing random circuit evolution 2–5 . Finally, we compare the experimental fidelity against that achieved by various approximate classical algorithms, and find that only the algorithm we introduce is able to keep pace with the experiment on the classical hardware we use. Our results enable a new model for evaluating the ability of both analogue and digital quantum devices to generate entanglement in the beyond-classically-exact regime, and highlight the evolving divide between quantum and classical systems.

Science & Technology - Other Topics↗
DOE OSTI2023

BigNeuron: a resource to benchmark and predict performance of algorithms for automated tracing of neurons in light microscopy datasets

BigNeuron is an open community bench-testing platform with the goal of setting open standards for accurate and fast automatic neuron tracing. We gathered a diverse set of image volumes across several species that is representative of the data obtained in many neuroscience laboratories interested in neuron tracing. Here, we report generated gold standard manual annotations for a subset of the available imaging datasets and quantified tracing quality for 35 automatic tracing algorithms. The goal of generating such a hand-curated diverse dataset is to advance the development of tracing algorithms and enable generalizable benchmarking. Together with image quality features, we pooled the data in an interactive web application that enables users and developers to perform principal component analysis, t-distributed stochastic neighbor embedding, correlation and clustering, visualization of imaging and tracing data, and benchmarking of automatic tracing algorithms in user-defined data subsets. The image quality metrics explain most of the variance in the data, followed by neuromorphological features related to neuron size. Furthermore, we observed that diverse algorithms can provide complementary information to obtain accurate results and developed a method to iteratively combine methods and generate consensus reconstructions. The consensus trees obtained provide estimates of the neuron structure ground truth that typically outperform single algorithms in noisy datasets. However, specific algorithms may outperform the consensus tree strategy in specific imaging conditions. Finally, to aid users in predicting the most accurate automatic tracing results without manual annotations for comparison, we used support vector machine regression to predict reconstruction quality given an image volume and a set of automatic tracings.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Energy landscapes from cryo-EM snapshots: a benchmarking study

Abstract Biomolecules undergo continuous conformational motions, a subset of which are functionally relevant. Understanding, and ultimately controlling biomolecular function are predicated on the ability to map continuous conformational motions, and identify the functionally relevant conformational trajectories. For equilibrium and near-equilibrium processes, function proceeds along minimum-energy pathways on one or more energy landscapes, because higher-energy conformations are only weakly occupied. With the growing interest in identifying functional trajectories, the need for reliable mapping of energy landscapes has become paramount. In response, various data-analytical tools for determining structural variability are emerging. A key question concerns the veracity with which each data-analytical tool can extract functionally relevant conformational trajectories from a collection of single-particle cryo-EM snapshots. Using synthetic data as an independently known ground truth, we benchmark the ability of four leading algorithms to determine biomolecular energy landscapes and identify the functionally relevant conformational paths on these landscapes. Such benchmarking is essential for systematic progress toward atomic-level movies of continuous biomolecular function.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2023

Federated benchmarking of medical artificial intelligence with MedPerf

Medical artificial intelligence (AI) has tremendous potential to advance healthcare by supporting and contributing to the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving both healthcare provider and patient experience. Unlocking this potential requires systematic, quantitative evaluation of the performance of medical AI models on large-scale, heterogeneous data capturing diverse patient populations. Here, to meet this need, we introduce MedPerf, an open platform for benchmarking AI models in the medical domain. MedPerf focuses on enabling federated evaluation of AI models, by securely distributing them to different facilities, such as healthcare organizations. This process of bringing the model to the data empowers each facility to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status and real-world deployment, our roadmap and, importantly, the use of MedPerf with multiple international institutions within cloud-based technology and on-premises scenarios. Finally, we welcome new contributions by researchers and organizations to further strengthen MedPerf as an open benchmarking platform.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2026

A curated benchmark for cofolding models on kinase conformational states

Abstract Protein kinases are critical drug targets, requiring therapeutics that can modulate their active and inactive conformational states. While cofolding models can generate global folds directly from kinase sequences and ligand SMILES strings, these models have not yet been tested on their ability to recover ligand-induced-fit conformational states of the kinase proteins. Here, we introduce KinConfBench, a curated benchmark of 2225 high-quality human kinase chains to evaluate the ability of four state-of-the-art cofolding models—Boltz-2, Chai-1, Protenix, and RoseTTAFold-All-Atom—to recover both canonical and rare conformational states. We show that geometric success metrics of a ligand pose in the active site do not correlate strongly with the correct kinase conformational state, motivating a new set of dynamical benchmarks for assessing cofolding models. While all four cofolding models achieve ~60–80% prediction accuracy for kinase conformational classification, they exhibit severe mode collapse when performing multiple inferences, show negligible structural diversity in sampling induced-fit motions, and display a prevalent “apo-drift” in which most cofolding models predominantly predict the kinase to be in its ligand-free state. Our results highlight that capturing ligand-induced protein conformational diversity, not just geometric fit, is critical for next-generation structure-based drug discovery.

Sun, Kunyang↗
DOE OSTI2018

On the geometric dependence of the molecular dipole polarizability in water: A benchmark study of higher-order electron correlation, basis set incompleteness error, core electron effects, and zero-point vibrational contributions

In this work, we investigate how geometric changes influence the static dipole polarizability ($α$) of a water molecule by explicitly computing the corresponding dipole polarizability surface (DPS) across 3125 total (1625 symmetry-unique) geometries using linear response coupled cluster theory including single, double, and triple excitations (LR-CCSDT) and the doubly augmented triple-$ζ$ basis set (d-aug-cc-pVTZ). Analytical formulae based on power series expansions of this $ab initio$ surface are generated using linear least-squares analysis and provide highly accurate estimates of this quantity as a function of molecular geometry (i.e., bond and angle variations) in a computationally tractable manner. An additional database, which consists of 25 representative molecular geometries and incorporates a more thorough treatment of both basis sets and core electron effects, is provided as a current benchmark for this quantity and the corresponding leading-order $C_6$ dispersion coefficient. This database has been utilized to assess the importance of these effects as well as the relative accuracy that can be obtained using several quantum chemical methods and a library of density functional approximations. In addition to high-level electron correlation methods (like CCSD) and our analytical least-squares formulae, we find that the SCAN0, PBE0, MN15, and B97-2 hybrid functionals yield the most accurate descriptions of the molecular polarizability tensor in H 2 O. Using first-order perturbation theory, we compute the zero-point vibrational correction to $α$ at the CCSDT/d-aug-cc-pVTZ level and find that this correction contributes approximately 3% to the isotropic ($α$ iso ) and nearly 50% to the anisotropic ($α$ aniso ) polarizability values. In doing so, we find that $α$ iso = 9.8307 bohr 3 , which is in excellent agreement with the experimental value of 9.83 ± 0.02 bohr 3 provided by Russell and Spackman. The DPS reported herein provides a benchmark-quality quantum mechanical estimate of this fundamental quantity of interest and should find extensive use in the development (and assessment) of next-generation force fields and machine-learning based approaches for modeling water in complex condensed-phase environments.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2019

A comprehensive dataset for the accelerated development and benchmarking of solar forecasting methods

We describe and release a comprehensive solar irradiance, imaging, and forecasting dataset. Our goal with this release is to provide standardized solar and meteorological datasets to the research community for the accelerated development and benchmarking of forecasting methods. The data consist of three years (2014–2016) of quality-controlled, 1-min resolution global horizontal irradiance and direct normal irradiance ground measurements in California. In addition, we provide overlapping data from commonly used exogenous variables, including sky images, satellite imagery, and Numerical Weather Prediction forecasts. We also include sample codes of baseline models for benchmarking of more elaborated models.

Remote sensing↗
DOE OSTI2020

Axisymmetric simulations of vertical displacement events in tokamaks: A benchmark of M3D-C 1 , NIMROD, and JOREK

A benchmark exercise for the modeling of vertical displacement events (VDEs) is presented and applied to the 3D nonlinear magnetohydrodynamic codes M3D-C 1 , JOREK, and NIMROD. The simulations are based on a vertically unstable NSTX equilibrium enclosed by an axisymmetric resistive wall with a rectangular cross section. A linear dependence of the linear VDE growth rates on the resistivity of the wall is recovered for sufficiently large wall conductivity and small temperatures in the open field line region. We report the benchmark results show good agreement between the VDE growth rates obtained from linear NIMROD and M3D-C 1 simulations and from the linear phase of axisymmetric nonlinear JOREK, NIMROD, and M3D-C 1 simulations. Axisymmetric nonlinear simulations of a full VDE performed with the three codes are compared, and an excellent agreement is found regarding the plasma location and plasma currents, as well as eddy and halo currents in the wall.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2020

Solvation at metal/water interfaces: An ab initio molecular dynamics benchmark of common computational approaches

Determining the influence of the solvent on electrochemical reaction energetics is a central challenge in our understanding of electrochemical interfaces. To date, it is unclear how well existing methods predict solvation energies at solid/liquid interfaces, since they cannot be assessed experimentally. Ab initio molecular dynamics (AIMD) simulations present a physically highly accurate, but also a very costly approach. In this work, we employ extensive AIMD simulations to benchmark solvation at charge-neutral metal/water interfaces against commonly applied continuum solvent models. We consider a variety of adsorbates including *CO, *CHO, *COH, *OCCHO, *OH, and *OOH on Cu, Au, and Pt facets solvated by water. The surfaces and adsorbates considered are relevant, among other reactions, to electrochemical CO 2 reduction and the oxygen redox reactions. We determine directional hydrogen bonds and steric water competition to be critical for a correct description of solvation at the metal/water interfaces. As a consequence, we find that the most frequently applied continuum solvation methods, which do not yet capture these properties, do not presently provide more accurate energetics over simulations in vacuum. We find most of the computed benchmark solvation energies to linearly scale with hydrogen bonding or competitive water adsorption, which strongly differ across surfaces. Furthermore, we determine solvation energies of adsorbates to be non-transferable between metal surfaces, in contrast to standard practice.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Gyrokinetic benchmark of the electron temperature-gradient instability in the pedestal region

Transport from turbulence driven by the electron temperature-gradient (ETG) instability is likely a major source of electron heat losses through the pedestal. Due to extreme gradients and strong shaping, ETG instabilities in the pedestal are distinct from those in the core, having, for example, multiple branches (toroidal and slab) in different wavenumber ranges. Due to its importance for pedestal transport, and its rather exotic character, a rigorous multi-code benchmarking exercise is imperative. In this work, we describe such an exercise, wherein we have carried out a detailed comparison of local linear pedestal ETG simulations using three gyrokinetic codes, CGYRO, GEM, and GENE and testing different geometric parameters (such as circular, Miller, and equilibrium EFIT geometry). The resulting linear frequencies, growth rates, and eigenfunctions show very good agreement between the codes in the three types of employed geometries. A nonlinear benchmark between CGYRO and GENE is also described, exhibiting good agreement (a maximum of 20% difference in the heat fluxes computed) at two locations in the pedestal. This lays the foundation for confidently modeling ETG turbulence in the pedestal

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2021

NENCI-2021. I. A large benchmark database of non-equilibrium non-covalent interactions emphasizing close intermolecular contacts

In this work, we present NENCI-2021, a benchmark database of ~8000 Non-Equilibirum Non-Covalent Interaction energies for a large and diverse selection of intermolecular complexes of biological and chemical relevance. To meet the growing demand for large and high-quality quantum mechanical data in the chemical sciences, NENCI-2021 starts with the 101 molecular dimers in the widely used S66 and S101 databases and extends the scope of these works by (i) including 40 cation–π and anion–π complexes, a fundamentally important class of non-covalent interactions that are found throughout nature and pose a substantial challenge to theory, and (ii) systematically sampling all 141 intermolecular potential energy surfaces (PESs) by simultaneously varying the intermolecular distance and intermolecular angle in each dimer. Designed with an emphasis on close contacts, the complexes in NENCI-2021 were generated by sampling seven intermolecular distances along each PES (ranging from 0.7× to 1.1× the equilibrium separation) and nine intermolecular angles per distance (five for each ion–π complex), yielding an extensive database of 7763 benchmark intermolecular interaction energies (E int ) obtained at the coupled-cluster with singles, doubles, and perturbative triples/complete basis set [CCSD(T)/CBS] level of theory. The E int values in NENCI-2021 span a total of 225.3 kcal/mol, ranging from -38.5 to +186.8 kcal/mol, with a mean (median) E int value of -1.06 kcal/mol (-2.39 kcal/mol). In addition, a wide range of intermolecular atom-pair distances are also present in NENCI-2021, where close intermolecular contacts involving atoms that are located within the so-called van der Waals envelope are prevalent—these interactions, in particular, pose an enormous challenge for molecular modeling and are observed in many important chemical and biological systems. A detailed symmetry-adapted perturbation theory (SAPT)- based energy decomposition analysis also confirms the diverse and comprehensive nature of the intermolecular binding motifs present in NENCI-2021, which now includes a significant number of primarily induction-bound dimers (e.g., cation–π complexes). NENCI-2021 thus spans all regions of the SAPT ternary diagram, thereby warranting a new four-category classification scheme that includes complexes primarily bound by electrostatics (3499), induction (700), dispersion (1372), or mixtures thereof (2192). A critical error analysis performed on a representative set of intermolecular complexes in NENCI-2021 demonstrates that the E int values provided herein have an average error of ±0.1 kcal/mol, even for complexes with strongly repulsive E int values, and maximum errors of ±0.2–0.3 kcal/mol (i.e., ~±1.0 kJ/mol) for the most challenging cases. For these reasons, we expect that NENCI-2021 will play an important role in the testing, training, and development of next-generation classical and polarizable force fields, density functional theory approximations, wavefunction theory methods, and machine learning based intra- and inter-molecular potentials.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

Preliminary benchmarks and analysis of boundary conditions in a trenched microstructured silicon radiation detector

Microstructured neutron detectors have the benefit of enhanced neutron detection efficiency as compared to planar devices, achieved by etching 6 LiF-filled trenches on the top surface of a silicon PIN diode. This sensor geometry results in a complex electric field distribution and depletion characteristics within the diode under reverse bias. For the first time on record, the effects of a fixed oxide charge on the microstructured device depletion characteristics and mobile carrier transport is investigated. Prototype detectors were fabricated with non-conformal surface doping. Capacitance voltage and current voltage measurements were performed for these prototypes and compared with COMSOL Multiphysics simulations. A spectral response from an 241Am alpha particle source was acquired and analyzed. It was found that monoenergetic alpha particles produce three prominent peaks in the pulse height spectrum output by the device. The peaks were confirmed by simulations to correlate with dead layers and incident trajectories into the microstructure. It was also found that significant differences in pulse rise time result, corresponding with events arriving in a low-field region in the fins and a high-field region in the bulk. Geant4 was utilized for radiation transport, interaction modeling, and benchmarking the spectral data. The results of this simulation work provide confidence in the ability to attain and benchmark electrical characteristics and spectral data for semiconductor radiation detectors employing complex microstructures.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2025

Phonon Olympics: Phonon property and lattice thermal conductivity benchmarking from open-source packages

Three widely used open-source packages for determining phonon properties and lattice thermal conductivities (ALAMODE, phono3py, and ShengBTE) are benchmarked by teams of expert users and the package developers. The phonons for Ge, RbBr, monolayer MoSe 2 , and AlN are modeled at zero temperature, and they scatter through three-phonon and phonon-isotope processes, with thermal conductivities obtained from the linearized Peierls–Boltzmann transport equation with input from density functional theory calculations. Over a wide range of temperatures, the thermal conductivities calculated by the teams fall within at most ±15% of their mean values for each of the four materials. The phonon frequencies, obtained from the harmonic force constants, do not show large differences between the calculations, indicating that the modal heat capacities and group velocities are not responsible for the thermal conductivity variations. It is the lifetimes associated with three-phonon scattering, obtained from the cubic force constants, that drive the variations. The many decisions required to calculate the cubic force constants (e.g., supercell size, atomic displacement, neighbor cutoff, and application of symmetries) make identification of the precise origin of the thermal conductivity variations challenging. The calculated thermal conductivities do not generally show agreement with experimental measurements, which is attributed to the limitations of the density functional theory calculations. Guidance for the development of best practices is provided, which will help to standardize protocols needed for building thermal conductivity databases. The results provide a baseline for future benchmarking of other packages and more advanced calculations.

McGaughey, Alan J. H. [Carnegie Mellon Univ., Pitt↗
DOE OSTI2026

Benchmarking core turbulence and transport predictions for an inductive compact tokamak reactor plasma

Motivated by the need for accurate, timely, and efficient calculations of plasma transport, predictions of plasma turbulence properties made using different TGLF saturation rules are benchmarked against corresponding predictions from linear and nonlinear gyrokinetic CGYRO simulations. This benchmarking is carried out using parameters taken from an inductive burning plasma scenario in a hypothetical compact high-field (R maj = 4 m, B T = 8 T) tokamak, lying in a much different regime of parameter space than either the TGLF calibration regime or current-day experiments. The core turbulent transport in this scenario is predicted to be dominated by ion temperature gradient (ITG) turbulence. In general, the ITG critical gradients predicted by various TGLF saturation rules are quite close to the CGYRO predictions. Both codes predict similar linear ITG growth rates and frequency spectra, as well as their scaling with R/L T i = −Rd ln(T i )/dr. However, TGLF systematically predicts unstable trapped-electron modes (TEMs) above k y ρ s ≃ 0.5 not seen by CGYRO for the same parameters, due to TGLF predicting a lower threshold in R/L T e than CGYRO for TEM onset. It is shown that for this scenario, nonlinear CGYRO simulations predict stiffer ITG turbulence than the TGLF SAT0 and SAT1 saturation rules, with energy fluxes close in magnitude and scaling with R/L T i to what is predicted by the SAT2 saturation rule. Self-consistent core profiles calculated using nonlinear CGYRO flux predictions and the PORTALS transport solver are shown to agree fairly well with corresponding predictions made using the TGLF SAT2 model, including a similar level of density peaking.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2026

Hofstadter butterfly and quantum transport benchmarks in PVA-exfoliated graphene heterostructures

Polymer exposure during van der Waals heterostructure fabrication is widely regarded as compromising the integrity of the electronic system required for hosting emergent quantum physics. This assumption has persisted largely because electronic benchmarking of heterostructures from polymer-exposed graphene has remained limited to only foundational transport metrics, such as mobility and charge inhomogeneity. Here, we challenge this assumption by establishing that graphene heterostructures produced by polyvinyl alcohol (PVA)-assisted exfoliation and encapsulated in hexagonal boron nitride using elevated-temperature lamination satisfy demanding quantum transport benchmarks. Beyond exhibiting ultra-high mobility and ballistic transport, these heterostructures yield quantum scattering times comparable to the best polymer-free devices. Most demanding of all, moiré superlattices from PVA-exposed graphene exhibit Hofstadter butterfly spectra, confirming spatially uniform interlayer coupling across the device area. These results establish that PVA exposure is compatible with low-disorder electronic systems, relaxing the trade-off between scalable fabrication and low-disorder quantum transport. This study motivates further development of polymer-assisted assembly with engineered residue-removal protocols.

36 MATERIALS SCIENCE↗