Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmark”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18

Diffusion Quantum Monte Carlo Benchmarking of Magnetic Moments in MnBi 2 Te 4

The intrinsically antiferromagnetic topological insulator, MnBi 2 Te 4 (MBT), has garnered significant attention recently due to its potential to host numerous exotic topological quantum states. Unfortunately, their consistent realization has been hindered by intrinsic antisite defects among the Mn and Bi sublattices. In this work, we establish Mn magnetization of pristine MBT through high level diffusion Monte Carlo calculations, which can serve as a precise starting point for various models to estimate antisite defect concentrations in actual MBT samples. The benchmark quality of DMC calculations is further identified from out model estimating antisite defect concentrations, which combines the benchmarked Mn magnetization with data from magnetic susceptibility and intermediate field magnetization measurements. This reproduces well Bi Mn and Mn Bi concentrations measured in the experiments. Here, we anticipate these theoretically based magnetic purity measures may be used as minimization targets in cycles of refinement to synthesize MBT with low antisite defect concentrations and more reproducible topological properties.

Defects↗

Guest-Host Interactions in Clathrate Hydrates: Benchmark MP2 and CCSD(T)/CBS Binding Energies of CH4, CO2 and H2S in (H2O)20 Cages

We present benchmark binding energies of naturally occurring gas molecules CH4, CO2, and H2S in the small cage, namely the pentagonal dodecahedron (512) (H2O)20, which is one of the constituent cages of the 3 major lattices (structures I, II and H) of clathrate hydrates. These weak interactions require higher levels of electron correlation and converge slowly with increasing basis set to the Complete Basis Set (CBS) limit, necessitating the use of large basis sets up to the augcc- pV5Z and subsequent correction for Basis Set Superposition Error (BSSE). For the host hollow (H2O)20 cages we have identified a most stable isomer with binding energy of -200.8 ± 2.1 kcal/mol at the CCSD(T)/CBS limit (-199.2 ± 0.5 kcal/mol at the MP2/CBS limit). Additionally, we report converged second order Moller-Plesset (MP2) CBS binding energies for the encapsulation of guests in the (H2O)20 cage of -4.3 ± 0.1 for CH4@(H2O)20, -6.6 ± 0.1 for CO2@(H2O)20 and -8.5 ± 0.1 kcal/mol for H2S@(H2O)20, respectively. For CH4@(H2O)20, exhibiting the weakest encapsulation affinity among the three, we report CCSD(T)/aug-cc-pVTZ binding energies and, based on them, a CCSD(T)/CBS estimate of -4.75 ± 0.1 kcal/mol. To the best of our knowledge, the CCSD(T)/aug-cc-pVTZ calculation for CH4@(H2O)20 is the largest one reported to date (168 valence electrons, 1978 basis functions and the correlation of 84 doubly occupied and 1873 virtual orbitals) and required a scalable implementation of the (T) module on 6144 nodes (350208 cores) of the “Cori” supercomputer at the National Energy Research Supercomputing Center (NERSC) for a total execution time of 195 minutes (for the (T) part). These efficient scalable implementations of highly correlated methods offer the capability to obtain long-lasting benchmarks of intermolecular interactions in complex systems. They also provide a path towards parametrizing classical potentials needed to study the dynamical and transport properties in these complex systems as well as assess the accuracy of lower scaling electronic structure methods such as Density Functional Theory (DFT) and MP2 including its spin-biased variants.

Heindel, Joseph↗

Barrier Heights for Diels–Alder Transition States Leading to Pentacyclic Adducts: A Benchmark Study of Crowded, Strained Transition States of Large Molecules

Theoretical characterization of reactions of complex molecules depends on providing consistent accuracy for the relative energies of intermediates and transition states. Here we employ the DLPNO-CCSD(T) method with core–valence correlation, large basis sets, and extrapolation to the CBS limit to provide benchmark values for Diels–Alder transition states leading to competitive strained pentacyclic adducts. We then used those benchmarks to test a diverse set of wave function and density functional methods for the absolute and relative barrier heights of these transition states. Our results show that only a few of the tested density functionals can predict the absolute barrier heights satisfactorily, although relative barrier heights are more accurate. The most accurate functionals tested are ωB97M-V, M11plus, ωB97X-V, PBE-D3(0), M11, and MN15 with MUDs from best estimates less than 3.0 kcal. Furthermore, these findings can guide selection of density functionals for future studies of crowded, strained transition states of large molecules.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Benchmarking quantum logic operations relative to thresholds for fault tolerance

Contemporary methods for benchmarking noisy quantum processors typically measure average error rates or process infidelities. However, thresholds for fault-tolerant quantum error correction are given in terms of worst-case error rates—defined via the diamond norm—which can differ from average error rates by orders of magnitude. One method for resolving this discrepancy is to randomize the physical implementation of quantum gates, using techniques like randomized compiling (RC). In this work, we use gate set tomography to perform precision characterization of a set of two-qubit logic gates to study RC on a superconducting quantum processor. We find that, under RC, gate errors are accurately described by a stochastic Pauli noise model without coherent errors, and that spatially correlated coherent errors and non-Markovian errors are strongly suppressed. We further show that the average and worst-case error rates are equal for randomly compiled gates, and measure a maximum worst-case error of 0.0197(3) for our gate set. Our results show that randomized benchmarks are a viable route to both verifying that a quantum processor’s error rates are below a fault-tolerance threshold, and to bounding the failure rates of near-term algorithms, if—and only if—gates are implemented via randomization methods which tailor noise.

97 MATHEMATICS AND COMPUTING↗

Quantum annealing for combinatorial optimization: a benchmarking study

Quantum annealing (QA) has the potential to significantly improve solution quality and reduce time complexity in solving combinatorial optimization problems compared to classical optimization methods. However, due to the limited number of qubits and their connectivity, the QA hardware did not show such an advantage over classical methods in past benchmarking studies. Recent advancements in QA with more than 5000 qubits, enhanced qubit connectivity, and the hybrid architecture promise to realize the quantum advantage. Here, we use a quantum annealer with state-of-the-art techniques and benchmark its performance against classical solvers. To compare their performance, we solve over 50 optimization problem instances represented by large and dense Hamiltonian matrices using quantum and classical solvers. The results demonstrate that a state-of-the-art quantum solver has higher accuracy (~0.013%) and a significantly faster problem-solving time (~6561×) than the best classical solver. Our results highlight the advantages of leveraging QA over classical counterparts, particularly in hybrid configurations, for achieving high accuracy and substantially reduced problem solving time in large-scale real-world optimization problems.

97 MATHEMATICS AND COMPUTING↗

Benchmarking highly entangled states on a 60-atom analogue quantum simulator

Abstract Quantum systems have entered a competitive regime in which classical computers must make approximations to represent highly entangled quantum states 1,2 . However, in this beyond-classically-exact regime, fidelity comparisons between quantum and classical systems have so far been limited to digital quantum devices 2–5 , and it remains unsolved how to estimate the actual entanglement content of experiments 6 . Here, we perform fidelity benchmarking and mixed-state entanglement estimation with a 60-atom analogue Rydberg quantum simulator, reaching a high-entanglement entropy regime in which exact classical simulation becomes impractical. Our benchmarking protocol involves extrapolation from comparisons against an approximate classical algorithm, introduced here, with varying entanglement limits. We then develop and demonstrate an estimator of the experimental mixed-state entanglement 6 , finding our experiment is competitive with state-of-the-art digital quantum devices performing random circuit evolution 2–5 . Finally, we compare the experimental fidelity against that achieved by various approximate classical algorithms, and find that only the algorithm we introduce is able to keep pace with the experiment on the classical hardware we use. Our results enable a new model for evaluating the ability of both analogue and digital quantum devices to generate entanglement in the beyond-classically-exact regime, and highlight the evolving divide between quantum and classical systems.

Science & Technology - Other Topics↗

BigNeuron: a resource to benchmark and predict performance of algorithms for automated tracing of neurons in light microscopy datasets

BigNeuron is an open community bench-testing platform with the goal of setting open standards for accurate and fast automatic neuron tracing. We gathered a diverse set of image volumes across several species that is representative of the data obtained in many neuroscience laboratories interested in neuron tracing. Here, we report generated gold standard manual annotations for a subset of the available imaging datasets and quantified tracing quality for 35 automatic tracing algorithms. The goal of generating such a hand-curated diverse dataset is to advance the development of tracing algorithms and enable generalizable benchmarking. Together with image quality features, we pooled the data in an interactive web application that enables users and developers to perform principal component analysis, t-distributed stochastic neighbor embedding, correlation and clustering, visualization of imaging and tracing data, and benchmarking of automatic tracing algorithms in user-defined data subsets. The image quality metrics explain most of the variance in the data, followed by neuromorphological features related to neuron size. Furthermore, we observed that diverse algorithms can provide complementary information to obtain accurate results and developed a method to iteratively combine methods and generate consensus reconstructions. The consensus trees obtained provide estimates of the neuron structure ground truth that typically outperform single algorithms in noisy datasets. However, specific algorithms may outperform the consensus tree strategy in specific imaging conditions. Finally, to aid users in predicting the most accurate automatic tracing results without manual annotations for comparison, we used support vector machine regression to predict reconstruction quality given an image volume and a set of automatic tracings.

97 MATHEMATICS AND COMPUTING↗

Energy landscapes from cryo-EM snapshots: a benchmarking study

Abstract Biomolecules undergo continuous conformational motions, a subset of which are functionally relevant. Understanding, and ultimately controlling biomolecular function are predicated on the ability to map continuous conformational motions, and identify the functionally relevant conformational trajectories. For equilibrium and near-equilibrium processes, function proceeds along minimum-energy pathways on one or more energy landscapes, because higher-energy conformations are only weakly occupied. With the growing interest in identifying functional trajectories, the need for reliable mapping of energy landscapes has become paramount. In response, various data-analytical tools for determining structural variability are emerging. A key question concerns the veracity with which each data-analytical tool can extract functionally relevant conformational trajectories from a collection of single-particle cryo-EM snapshots. Using synthetic data as an independently known ground truth, we benchmark the ability of four leading algorithms to determine biomolecular energy landscapes and identify the functionally relevant conformational paths on these landscapes. Such benchmarking is essential for systematic progress toward atomic-level movies of continuous biomolecular function.

59 BASIC BIOLOGICAL SCIENCES↗

Federated benchmarking of medical artificial intelligence with MedPerf

Medical artificial intelligence (AI) has tremendous potential to advance healthcare by supporting and contributing to the evidence-based practice of medicine, personalizing patient treatment, reducing costs, and improving both healthcare provider and patient experience. Unlocking this potential requires systematic, quantitative evaluation of the performance of medical AI models on large-scale, heterogeneous data capturing diverse patient populations. Here, to meet this need, we introduce MedPerf, an open platform for benchmarking AI models in the medical domain. MedPerf focuses on enabling federated evaluation of AI models, by securely distributing them to different facilities, such as healthcare organizations. This process of bringing the model to the data empowers each facility to assess and verify the performance of AI models in an efficient and human-supervised process, while prioritizing privacy. We describe the current challenges healthcare and AI communities face, the need for an open platform, the design philosophy of MedPerf, its current implementation status and real-world deployment, our roadmap and, importantly, the use of MedPerf with multiple international institutions within cloud-based technology and on-premises scenarios. Finally, we welcome new contributions by researchers and organizations to further strengthen MedPerf as an open benchmarking platform.

60 APPLIED LIFE SCIENCES↗

A curated benchmark for cofolding models on kinase conformational states

Abstract Protein kinases are critical drug targets, requiring therapeutics that can modulate their active and inactive conformational states. While cofolding models can generate global folds directly from kinase sequences and ligand SMILES strings, these models have not yet been tested on their ability to recover ligand-induced-fit conformational states of the kinase proteins. Here, we introduce KinConfBench, a curated benchmark of 2225 high-quality human kinase chains to evaluate the ability of four state-of-the-art cofolding models—Boltz-2, Chai-1, Protenix, and RoseTTAFold-All-Atom—to recover both canonical and rare conformational states. We show that geometric success metrics of a ligand pose in the active site do not correlate strongly with the correct kinase conformational state, motivating a new set of dynamical benchmarks for assessing cofolding models. While all four cofolding models achieve ~60–80% prediction accuracy for kinase conformational classification, they exhibit severe mode collapse when performing multiple inferences, show negligible structural diversity in sampling induced-fit motions, and display a prevalent “apo-drift” in which most cofolding models predominantly predict the kinase to be in its ligand-free state. Our results highlight that capturing ligand-induced protein conformational diversity, not just geometric fit, is critical for next-generation structure-based drug discovery.

Sun, Kunyang↗

Axisymmetric simulations of vertical displacement events in tokamaks: A benchmark of M3D-C 1 , NIMROD, and JOREK

A benchmark exercise for the modeling of vertical displacement events (VDEs) is presented and applied to the 3D nonlinear magnetohydrodynamic codes M3D-C 1 , JOREK, and NIMROD. The simulations are based on a vertically unstable NSTX equilibrium enclosed by an axisymmetric resistive wall with a rectangular cross section. A linear dependence of the linear VDE growth rates on the resistivity of the wall is recovered for sufficiently large wall conductivity and small temperatures in the open field line region. We report the benchmark results show good agreement between the VDE growth rates obtained from linear NIMROD and M3D-C 1 simulations and from the linear phase of axisymmetric nonlinear JOREK, NIMROD, and M3D-C 1 simulations. Axisymmetric nonlinear simulations of a full VDE performed with the three codes are compared, and an excellent agreement is found regarding the plasma location and plasma currents, as well as eddy and halo currents in the wall.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Solvation at metal/water interfaces: An ab initio molecular dynamics benchmark of common computational approaches

Determining the influence of the solvent on electrochemical reaction energetics is a central challenge in our understanding of electrochemical interfaces. To date, it is unclear how well existing methods predict solvation energies at solid/liquid interfaces, since they cannot be assessed experimentally. Ab initio molecular dynamics (AIMD) simulations present a physically highly accurate, but also a very costly approach. In this work, we employ extensive AIMD simulations to benchmark solvation at charge-neutral metal/water interfaces against commonly applied continuum solvent models. We consider a variety of adsorbates including *CO, *CHO, *COH, *OCCHO, *OH, and *OOH on Cu, Au, and Pt facets solvated by water. The surfaces and adsorbates considered are relevant, among other reactions, to electrochemical CO 2 reduction and the oxygen redox reactions. We determine directional hydrogen bonds and steric water competition to be critical for a correct description of solvation at the metal/water interfaces. As a consequence, we find that the most frequently applied continuum solvation methods, which do not yet capture these properties, do not presently provide more accurate energetics over simulations in vacuum. We find most of the computed benchmark solvation energies to linearly scale with hydrogen bonding or competitive water adsorption, which strongly differ across surfaces. Furthermore, we determine solvation energies of adsorbates to be non-transferable between metal surfaces, in contrast to standard practice.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Gyrokinetic benchmark of the electron temperature-gradient instability in the pedestal region

Transport from turbulence driven by the electron temperature-gradient (ETG) instability is likely a major source of electron heat losses through the pedestal. Due to extreme gradients and strong shaping, ETG instabilities in the pedestal are distinct from those in the core, having, for example, multiple branches (toroidal and slab) in different wavenumber ranges. Due to its importance for pedestal transport, and its rather exotic character, a rigorous multi-code benchmarking exercise is imperative. In this work, we describe such an exercise, wherein we have carried out a detailed comparison of local linear pedestal ETG simulations using three gyrokinetic codes, CGYRO, GEM, and GENE and testing different geometric parameters (such as circular, Miller, and equilibrium EFIT geometry). The resulting linear frequencies, growth rates, and eigenfunctions show very good agreement between the codes in the three types of employed geometries. A nonlinear benchmark between CGYRO and GENE is also described, exhibiting good agreement (a maximum of 20% difference in the heat fluxes computed) at two locations in the pedestal. This lays the foundation for confidently modeling ETG turbulence in the pedestal

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

NENCI-2021. I. A large benchmark database of non-equilibrium non-covalent interactions emphasizing close intermolecular contacts

In this work, we present NENCI-2021, a benchmark database of ~8000 Non-Equilibirum Non-Covalent Interaction energies for a large and diverse selection of intermolecular complexes of biological and chemical relevance. To meet the growing demand for large and high-quality quantum mechanical data in the chemical sciences, NENCI-2021 starts with the 101 molecular dimers in the widely used S66 and S101 databases and extends the scope of these works by (i) including 40 cation–π and anion–π complexes, a fundamentally important class of non-covalent interactions that are found throughout nature and pose a substantial challenge to theory, and (ii) systematically sampling all 141 intermolecular potential energy surfaces (PESs) by simultaneously varying the intermolecular distance and intermolecular angle in each dimer. Designed with an emphasis on close contacts, the complexes in NENCI-2021 were generated by sampling seven intermolecular distances along each PES (ranging from 0.7× to 1.1× the equilibrium separation) and nine intermolecular angles per distance (five for each ion–π complex), yielding an extensive database of 7763 benchmark intermolecular interaction energies (E int ) obtained at the coupled-cluster with singles, doubles, and perturbative triples/complete basis set [CCSD(T)/CBS] level of theory. The E int values in NENCI-2021 span a total of 225.3 kcal/mol, ranging from -38.5 to +186.8 kcal/mol, with a mean (median) E int value of -1.06 kcal/mol (-2.39 kcal/mol). In addition, a wide range of intermolecular atom-pair distances are also present in NENCI-2021, where close intermolecular contacts involving atoms that are located within the so-called van der Waals envelope are prevalent—these interactions, in particular, pose an enormous challenge for molecular modeling and are observed in many important chemical and biological systems. A detailed symmetry-adapted perturbation theory (SAPT)- based energy decomposition analysis also confirms the diverse and comprehensive nature of the intermolecular binding motifs present in NENCI-2021, which now includes a significant number of primarily induction-bound dimers (e.g., cation–π complexes). NENCI-2021 thus spans all regions of the SAPT ternary diagram, thereby warranting a new four-category classification scheme that includes complexes primarily bound by electrostatics (3499), induction (700), dispersion (1372), or mixtures thereof (2192). A critical error analysis performed on a representative set of intermolecular complexes in NENCI-2021 demonstrates that the E int values provided herein have an average error of ±0.1 kcal/mol, even for complexes with strongly repulsive E int values, and maximum errors of ±0.2–0.3 kcal/mol (i.e., ~±1.0 kJ/mol) for the most challenging cases. For these reasons, we expect that NENCI-2021 will play an important role in the testing, training, and development of next-generation classical and polarizable force fields, density functional theory approximations, wavefunction theory methods, and machine learning based intra- and inter-molecular potentials.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Preliminary benchmarks and analysis of boundary conditions in a trenched microstructured silicon radiation detector

Microstructured neutron detectors have the benefit of enhanced neutron detection efficiency as compared to planar devices, achieved by etching 6 LiF-filled trenches on the top surface of a silicon PIN diode. This sensor geometry results in a complex electric field distribution and depletion characteristics within the diode under reverse bias. For the first time on record, the effects of a fixed oxide charge on the microstructured device depletion characteristics and mobile carrier transport is investigated. Prototype detectors were fabricated with non-conformal surface doping. Capacitance voltage and current voltage measurements were performed for these prototypes and compared with COMSOL Multiphysics simulations. A spectral response from an 241Am alpha particle source was acquired and analyzed. It was found that monoenergetic alpha particles produce three prominent peaks in the pulse height spectrum output by the device. The peaks were confirmed by simulations to correlate with dead layers and incident trajectories into the microstructure. It was also found that significant differences in pulse rise time result, corresponding with events arriving in a low-field region in the fins and a high-field region in the bulk. Geant4 was utilized for radiation transport, interaction modeling, and benchmarking the spectral data. The results of this simulation work provide confidence in the ability to attain and benchmark electrical characteristics and spectral data for semiconductor radiation detectors employing complex microstructures.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Phonon Olympics: Phonon property and lattice thermal conductivity benchmarking from open-source packages

Three widely used open-source packages for determining phonon properties and lattice thermal conductivities (ALAMODE, phono3py, and ShengBTE) are benchmarked by teams of expert users and the package developers. The phonons for Ge, RbBr, monolayer MoSe 2 , and AlN are modeled at zero temperature, and they scatter through three-phonon and phonon-isotope processes, with thermal conductivities obtained from the linearized Peierls–Boltzmann transport equation with input from density functional theory calculations. Over a wide range of temperatures, the thermal conductivities calculated by the teams fall within at most ±15% of their mean values for each of the four materials. The phonon frequencies, obtained from the harmonic force constants, do not show large differences between the calculations, indicating that the modal heat capacities and group velocities are not responsible for the thermal conductivity variations. It is the lifetimes associated with three-phonon scattering, obtained from the cubic force constants, that drive the variations. The many decisions required to calculate the cubic force constants (e.g., supercell size, atomic displacement, neighbor cutoff, and application of symmetries) make identification of the precise origin of the thermal conductivity variations challenging. The calculated thermal conductivities do not generally show agreement with experimental measurements, which is attributed to the limitations of the density functional theory calculations. Guidance for the development of best practices is provided, which will help to standardize protocols needed for building thermal conductivity databases. The results provide a baseline for future benchmarking of other packages and more advanced calculations.

McGaughey, Alan J. H. [Carnegie Mellon Univ., Pitt↗

Benchmarking core turbulence and transport predictions for an inductive compact tokamak reactor plasma

Motivated by the need for accurate, timely, and efficient calculations of plasma transport, predictions of plasma turbulence properties made using different TGLF saturation rules are benchmarked against corresponding predictions from linear and nonlinear gyrokinetic CGYRO simulations. This benchmarking is carried out using parameters taken from an inductive burning plasma scenario in a hypothetical compact high-field (R maj = 4 m, B T = 8 T) tokamak, lying in a much different regime of parameter space than either the TGLF calibration regime or current-day experiments. The core turbulent transport in this scenario is predicted to be dominated by ion temperature gradient (ITG) turbulence. In general, the ITG critical gradients predicted by various TGLF saturation rules are quite close to the CGYRO predictions. Both codes predict similar linear ITG growth rates and frequency spectra, as well as their scaling with R/L T i = −Rd ln(T i )/dr. However, TGLF systematically predicts unstable trapped-electron modes (TEMs) above k y ρ s ≃ 0.5 not seen by CGYRO for the same parameters, due to TGLF predicting a lower threshold in R/L T e than CGYRO for TEM onset. It is shown that for this scenario, nonlinear CGYRO simulations predict stiffer ITG turbulence than the TGLF SAT0 and SAT1 saturation rules, with energy fluxes close in magnitude and scaling with R/L T i to what is predicted by the SAT2 saturation rule. Self-consistent core profiles calculated using nonlinear CGYRO flux predictions and the PORTALS transport solver are shown to agree fairly well with corresponding predictions made using the TGLF SAT2 model, including a similar level of density peaking.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Hofstadter butterfly and quantum transport benchmarks in PVA-exfoliated graphene heterostructures

Polymer exposure during van der Waals heterostructure fabrication is widely regarded as compromising the integrity of the electronic system required for hosting emergent quantum physics. This assumption has persisted largely because electronic benchmarking of heterostructures from polymer-exposed graphene has remained limited to only foundational transport metrics, such as mobility and charge inhomogeneity. Here, we challenge this assumption by establishing that graphene heterostructures produced by polyvinyl alcohol (PVA)-assisted exfoliation and encapsulated in hexagonal boron nitride using elevated-temperature lamination satisfy demanding quantum transport benchmarks. Beyond exhibiting ultra-high mobility and ballistic transport, these heterostructures yield quantum scattering times comparable to the best polymer-free devices. Most demanding of all, moiré superlattices from PVA-exposed graphene exhibit Hofstadter butterfly spectra, confirming spatially uniform interlayer coupling across the device area. These results establish that PVA exposure is compatible with low-disorder electronic systems, relaxing the trade-off between scalable fabrication and low-disorder quantum transport. This study motivates further development of polymer-assisted assembly with engineered residue-removal protocols.

36 MATERIALS SCIENCE↗