Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

25-Pin metallic fuel performance benchmark case based on the EBR-II X430 experiments series

A metallic fuel benchmark case was developed for the fuel performance code BISON based on 25 uranium-zirconium and uranium-plutonium-zirconium pins of the Experimental Breeder Reactor II X430 experiment series. Results of the benchmarks were compared with measurements and calculations made at the time of the experiment as well as subsequent measurements reported in 2019. The comparisons were used to quantify the accuracy of the BISON predictions and to identify patterns in the BISON differences.BISON predicted burnup, plenum pressure, and fission gas release fractions accurately. BISON temperature predictions were somewhat cooler than the temperatures determined at the time of the experiment but appeared to be reasonably accurate considering uncertainties in the legacy temperature calculations, uncertainties in the legacy linear heat rate calculations, and the high sensitivities of the BISON-predicted temperatures to BISON inputs. Fuel axial elongation predictions had errors correlated to fuel composition; BISON tended to underpredict the elongation of binary fuels and overpredict the elongation of ternary fuels. BISON cladding radial dilation predictions were also significantly lower than legacy PIE measurements.Recommendations were made to improve the BISON fuel gaseous swelling model to account for fuel composition, to add additional capabilities to the coolant channel temperature model to ease benchmark development, and to continue developing benchmark cases based on a wide range of experiments in several reactors. Once a wide array of benchmarks is developed, an attempt can be made to enhance or calibrate BISON models to improve the cladding dilation predictions.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Extending quantum-mechanical benchmark accuracy to biological ligand-pocket interactions

Predicting the binding affinity of ligands to protein pockets is key in the drug design pipeline. The flexibility of ligand-pocket motifs arises from a range of attractive and repulsive electronic interactions during binding. Accurately accounting for all interactions requires robust quantum-mechanical (QM) benchmarks, which are scarce for ligand-pocket systems. Additionally, disagreement between “gold standard” Coupled Cluster (CC) and Quantum Monte Carlo (QMC) methods casts doubt on many benchmarks for larger non-covalent systems. We introduce the “QUantum Interacting Dimer” (QUID) benchmark framework containing 170 non-covalent (non-)equilibrium systems modeling chemically and structurally diverse ligand-pocket motifs. Symmetry-adapted perturbation theory shows that QUID broadly covers non-covalent binding motifs and energetic contributions. Robust binding energies are obtained using complementary CC and QMC methods, achieving agreement of 0.5 kcal/mol. The benchmark data analysis reveals that several dispersion-inclusive density functional approximations provide accurate energy predictions, though their atomic van der Waals forces differ in magnitude and orientation. Contrarily, semiempirical methods and empirical force fields require improvements in capturing non-covalent interactions (NCIs) for out-of-equilibrium geometries. The wide span of NCIs, highly accurate interaction energies, and analysis of molecular properties take QUID beyond the “gold standard” for QM benchmarks of ligand-protein systems.

Puleva, Mirela [University of Luxembourg, Luxembou↗

Quantum chemistry as a benchmark for near-term quantum computers

We present a quantum chemistry benchmark for noisy intermediate-scale quantum computers that leverages the variational quantum eigensolver, active-space reduction, a reduced unitary coupled cluster ansatz, and reduced density purification as error mitigation. We demonstrate this benchmark using 4 of the available qubits on the 20-qubit IBM Tokyo and 16-qubit Rigetti Aspen processors via the simulation of alkali metal hydrides (NaH, KH, RbH), with accuracy of the computed ground state energy serving as the primary benchmark metric. We further parameterize this benchmark suite on the trial circuit type, the level of symmetry reduction, and error mitigation strategies. Our results demonstrate the characteristically high noise level present in near-term superconducting hardware, but provide a relevant baseline for future improvement of the underlying hardware, and a means for comparison across near-term hardware types. We also demonstrate how to reduce the noise in post processing with specific error mitigation techniques. Particularly, the adaptation of McWeeny purification of noisy density matrices dramatically improves accuracy of quantum computations, which, along with adjustable active space, significantly extends the range of accessible molecular systems. We demonstrate that for specific benchmark settings and a selected range of problems, the accuracy metric can reach chemical accuracy when computing over the cloud on certain quantum computers.

97 MATHEMATICS AND COMPUTING↗

A robust benchmark for detection of germline large deletions and insertions

New technologies and analysis methods are enabling genomic structural variants (SVs) to be detected with ever-increasing accuracy, resolution and comprehensiveness. To help translate these methods to routine research and clinical practice, we developed a sequence-resolved benchmark set for identification of both false-negative and false-positive germline large insertions and deletions. To create this benchmark for a broadly consented son in a Personal Genome Project trio with broadly available cells and DNA, the Genome in a Bottle Consortium integrated 19 sequence-resolved variant calling methods from diverse technologies. The final benchmark set contains 12,745 isolated, sequence-resolved insertion (7,281) and deletion (5,464) calls ≥50 base pairs (bp). The Tier 1 benchmark regions, for which any extra calls are putative false positives, cover 2.51 Gbp and 5,262 insertions and 4,095 deletions supported by ≥1 diploid assembly. We demonstrate that the benchmark set reliably identifies false negatives and false positives in high-quality SV callsets from short-, linked- and long-read sequencing and optical mapping.

59 BASIC BIOLOGICAL SCIENCES↗

Estimating Code Biases for Criticality Safety Applications with Few Relevant Benchmarks

Criticality safety analyses rely on the availability of relevant benchmark experiments to determine justifiable margins of subcriticality. When a target application lacks neutronically similar benchmark experiments, validation studies must provide justification to the regulator that the impact of modeling and simulation limitations is well understood for the application and often must provide additional subcritical margin to ensure safe operating conditions. This study estimated the computational bias in the critical eigenvalue for several criticality safety applications supported by only a few relevant benchmark experiments. The accuracy of the following three methods for predicting computational biases was evaluated: the Upper Subcritical Limit STATisticS (USLSTATS) trending analysis method; the Whisper nonparametric method; and TSURFER, which is based on the generalized linear least-squares technique. Here, these methods were also applied to estimate computational biases and recommended upper subcriticality limits for several critical experiments with known biases and for several cases from a blind benchmark study. The methods are evaluated based on both the accuracy of their predicted computation bias and upper subcriticality limit estimates, as well as on the consistency of the methods’ estimates, as the model parameters, covariance data libraries, and set of available benchmark data were varied. Data assimilation methods typically have not been used for criticality safety licensing activities, and this study explores a methodology to address concerns regarding the reliability of such methods in criticality safety bias prediction applications.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Performance Portability Evaluation of OpenCL Benchmarks across Intel and NVIDIA Platforms

We evaluate the capabilities of vendor-provided OpenCL implementations for performance portability across multiple computing platforms. The Rodinia benchmark suite is used for this evaluation. We apply the metric defined by Pennycook et al., and we use roofline efficiency from the Roofline performance model as the "performance efficiency" in the metric's definition. We found that the delivered performance portability is similar for several benchmarks, even if the roofline-based performance efficiencies across platforms are very different among the benchmarks. To help distinguish between these instances, we extend the metric by adding the standard deviation of the performance efficiencies for each benchmark. We argue that the standard deviation gives additional insight into performance portability assessment since it adds the performance variability across platforms. Additionally, we discuss the challenges to measure performance portability associated with algorithms and system software. In terms of algorithms, we need to carefully construct the benchmarks and appropriately use the concurrency available on a platform. In terms of system software, we depend on the vendor performance tools to support the desired programming model and runtime to be able to measure the metrics of interest.

GPU↗

miniVite: A Graph Analytics Benchmarking Tool for Massively Parallel Systems

Benchmarking of high performance computing sys-tems can help provide critical insights for ef?cient design of computing systems and software applications. Although a large number of tools for benchmarking exist, there is a lack of rep-resentative benchmarks for the class of irregular computations as exempli?ed by graph analytics. In this paper, we propose miniVite as a representative graph analytics benchmark tool to test a variety of distributed-memory systems. Graph clustering, popularly known as community detection, is a prototypical graph operation used in numerous scienti?c computing and analytics applications. The goal of clustering is to partition a graph into clusters (or communities) such that each cluster consists of vertices that are densely connected within the cluster and sparsely connected to the rest of the graph. Modu-larity optimization is a popular technique for identifying clusters in a graph. Ef?cient parallelization of modularity optimization-based algorithms is challenging. One successful approach was conceived in Vite, a distributed-memory implementation of the Louvain algorithm that incorporates several heuristics. We introduce miniVite as a representative but simpli?ed variant of Vite, to serve as a prototypical graph analytics benchmarking tool. Unlike other graph-based methods such as breadth-?rst search and betweenness centrality, miniVite represents highly complex computational patterns stressing a variety of system features, which can provide crucial insight for co-design of future computing systems.

community detection, parallel, MPI↗

WfBench: Automated Generation of Scientific Workflow Benchmarks

The prevalence of scientific workflows with high computational demands calls for their execution on various distributed computing platforms, including large-scale leadership-class high-performance computing (HPC) clusters. To handle the deployment, monitoring, and optimization of workflow executions, many workflow systems have been developed over the past decade. There is a need for workflow benchmarks that can be used to evaluate the performance of workflow systems on current and future software stacks and hardware platforms.We present a generator of realistic workflow benchmark specifications that can be translated into benchmark code to be executed with current workflow systems. Our approach generates workflow tasks with arbitrary performance characteristics (CPU, memory, and I/O usage) and with realistic task dependency structures based on those seen in production workflows. We present experimental results that show that our approach generates benchmarks that are representative of production workflows, and conduct a case study to demonstrate the use and usefulness of our generated benchmarks to evaluate the performance of workflow systems under different configuration scenarios.

Coleman, Taina↗

FireHose Streaming Benchmarks

The FireHose Streaming Benchmarks are a suite of stream-processing benchmarks defined to enable comparison of streaming software and hardware, both quantitatively vis-a-vis the rate at which they can process data, and qualitatively by judging the effort involved to implement and run the benchmarks. Each benchmark has two parts. The first is a generator which produces and outputs datums at a high rate in a specific format. The second is an analytic which reads the stream of datums and is required to perform a well-defined calculation on the collection of datums, typically to find anomalous datums that have been created in the stream by the generator. The FireHose suite provides code for the generators, sample code for the analytics (which users are free to re-implement in their own custom frameworks), and a precise definition of each benchmark calculation.

Plimpton, Steve↗

Assessment of four strain energy decomposition methods for phase field fracture models using quasi-static and dynamic benchmark cases

Abstract Strain energy decomposition methods in phase field fracture models separate strain energy that contributes to fracture from that which does not. However, various decomposition methods have been proposed in the literature, and it can be difficult to determine an appropriate method for a given problem. The goal of this work is to facilitate the choice of strain decomposition method by assessing the performance of three existing methods (spectral decomposition of the stress or the strain and deviatoric decomposition of the strain) and one new method (deviatoric decomposition of the stress) with several benchmark problems. In each benchmark problem, we compare the performance of the four methods using both qualitative and quantitative metrics. In the first benchmark, we compare the predicted mechanical behavior of cracked material. We then use four quasi-static benchmark cases: a single edge notched tension test, a single edge notched shear test, a three-point bending test, and a L-shaped panel test. Finally, we use two dynamic benchmark cases: a dynamic tensile fracture test and a dynamic shear fracture test. All four methods perform well in tension, the two spectral methods perform better in compression and with mixed mode (though the stress spectral method performs the best), and all the methods show minor issues in at least one of the shear cases. In general, whether the strain or the stress is decomposed does not have a significant impact on the predicted behavior.

Zhang, Shuaifang↗

Benchmarking Non-Hardware Balance-of-System (Soft) Costs for U.S. Photovoltaic Systems Using a Bottom-Up Approach and Installer Survey

This report presents results from the first U.S. Department of Energy (DOE) sponsored, bottom-up data-collection and analysis of non-hardware balance-of-system costs—often referred to as “business process” or “soft” costs—for residential and commercial photovoltaic (PV) systems. Annual expenditure and labor-hour-productivity data are analyzed to benchmark 2010 soft costs related to the DOE priority areas of (1) customer acquisition; (2) permitting, inspection, and interconnection; (3) installation labor; and (4) installer labor for arranging third-party financing. Annual expenditure and labor-hour data were collected from 87 PV installers. After eliminating outliers, the survey sample consists of 75 installers, representing approximately 13% of all residential PV installations and 4% of all commercial installations added in 2010. Including assumed permitting fees, in 2010 the average soft costs benchmarked in this analysis total $1.50/W for residential systems (ranging from $0.66/W to $1.66/W between the 20th and 80th percentiles). For commercial systems, the median 2010 benchmarked soft costs (including assumed permitting fees) are $0.99/W for systems smaller than 250 kW (ranging from $0.51/W to $1.45/W between the 20th and 80th percentiles) and $0.25/W for systems larger than 250 kW (ranging from $0.17/W to $0.78/W between the 20th and 80th percentiles). Additional soft costs not benchmarked in the present analysis (e.g., installer profit, overhead, financing, and contracting) are significant and would add to these figures. The survey results provide a benchmark for measuring—and helping to accelerate—progress over the next decade toward achieving the DOE SunShot Initiative’s soft-cost-reduction targets. We conclude that the selected non-hardware business processes add considerable cost to U.S. PV systems, constituting 23% of residential PV system price, 17% of small commercial system price, and 5% of large commercial system price (in 2010). These processes present significant opportunities for further cost reductions and labor-productivity gains.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Comprehensive Benchmark Suite for Simulation of Particle Laden Flows Using the Discrete Element Method with Performance Profiles from the Multiphase Flow with Interface eXchanges (MFiX) Code

Five benchmark problems are developed and simulated with the computational fluid dynamics and discrete element model code MFiX. The benchmark problems span dilute and dense regimes, consider statistically homogeneous and inhomogeneous (both clusters and bubbles) particle concentrations and a range of particle and fluid dynamic computational loads. Several variations of the benchmark problems are also discussed to extend the computational phase space to cover granular (particles only), bidisperse and heat transfer cases. A weak scaling analysis is performed for each benchmark problem and, in most cases, the scalability of the code appears reasonable up to approx. 103 cores. Profiling of the benchmark problems indicate that the most substantial computational time is being spent on particle-particle force calculations, drag force calculations and interpolating between discrete particle and continuum fields. Hardware performance analysis was also carried out showing significant Level 2 cache miss ratios and a rather low degree of vectorization. These results are intended to serve as a baseline for future developments to the code as well as a preliminary indicator of where to best focus performance optimizations.

97 MATHEMATICS AND COMPUTING↗

Benchmark Analysis of the HTR-10 with the MAMMOTH Reactor Physics Application

This report documents a set of benchmarks used to validate the Serpent and MAMMOTH neutron transport models of the HTR-10 reactor developed at Idaho National Laboratory. The high-fidelity Serpent Monte Carlo models of the HTR-10 critical and full core configurations include both random, discrete distributions of TRISO particles in the pebbles and random, discrete pebble distributions in the pebble bed core. The Serpent results agree very well with the critical and control rod worth measurements provided in the International Re- actor Physics Experiment Evaluation Project (IRPhEP) report if the ENDF/B-VII.r1 data- set is used. These Serpent models are subsequently used to prepare cross sections and flux tallies in a 10 coarse energy group structure for the MAMMOTH Reactor Physics MOOSE- based application. We find that MAMMOTH can reproduce the Monte Carlo solution for the HTR-10 reactor by using the PJFNK-SPH equivalence method. In all cases studied, the MAMMOTH results are within 120 pcm of the Monte Carlo reference calculation and the maximum errors in neutron absorption and generation rates are within 0.536% and 0.215% from the reference calculation, respectively. A set of benchmark exercises from the International Atomic Energy Agency (IAEA) HTR-10 benchmark in IAEA-TECDOC-1382 were conducted including parts B1 (critical), B2 (temperature coefficient) and B3 (control rod worth). The results we obtain for the critical core are in excellent agreement with the expected value and we confirm that with ENDF/B-VII.r1 data we match the critical eigen- value keff=1 to within 100 pcm. Our temperature coefficient calculation is consistent with the VSOP results generated by the German participants. The Chinese and South African VSOP models produce lower temperature coefficients, a fact that was noted in the original IAEA benchmark, but for which no explanation was provided. Both a full and single control rod worth calculations with Serpent and MAMMOTH agree well with the results from the IAEA benchmark. Finally, and most importantly, this work demonstrates that MAM- MOTH can recover the Monte Carlo high-fidelity solutions to an excellent accuracy using the PJFNK-SPH method. This sets the stage for the ultimate goal of using MAMMOTH in transient, multi-physics scenarios where correction of the cross sections is obtained for steady-state conditions and then used in the transient scenario for which no Monte-Carlo solution is available.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Creation of a Benchmark Suite for SNL Nuclear Criticality Safety Code Validation (Volume 1 - MCNP)

The Nuclear Criticality Safety Program at Sandia National Laboratories has developed a suite of benchmark problems to be utilized for validation of MCNP6 Version 1.0 with ENDF/B-VII Release 1 cross sections. The benchmark suite covers a broad range of fissile material types, material forms, moderators, reflectors, and neutron energy spectra. It is anticipated that this benchmark suite will cover the vast majority of critical safety applications at SNL. The benchmark suite establishes a Bias and Bias Uncertainty for use in criticality safety analyses. In addition, the Bias and Bias Uncertainty value derived from the benchmark suite using the traditional SNL NCS methodology is demonstrated to be equivalent to the more robust statistical techniques used at many DOE sites.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Reactor Physics Experiment NEA Benchmark Possibilities for Measurements at ORCEF and Other DOE Facilities

This report documents a wide variety of critical and subcritical reactor physics measurements that could be included in the Nuclear Energy Agency (NEA) International Handbook of Evaluated Reactor Physics Benchmark Experiments. The experiments were performed by John T. Mihalczo and coworkers at many (8) DOE facilities including the Oak Ridge National Laboratory Critical Experiments Facility (ORCEF between 1958 and 1975), Los Alamos National Laboratory, Pacific Northwest National Laboratory, and Babcock and Wilcox at Lynchburg, Virginia, and the Y-12 National Security Complex (the latter between 1975 and up to 2006). This report lists the various measurement programs and a limited description of the reactor physics measurement that were performed in each that could be benchmarks. Some criticality safety measurements at these facilities have already been documented in the NEA International Handbook of evaluated Criticality Safety Benchmark Experiments. Some of these reactor physics measurements have also been documented in the literature and this document give a complete list of references for which John T. Mihalczo was an author or co-author. The experiments are divided into four classes: critical experiments only, critical and subsequent subcritical experiments, subcritical experiments at critical facilities, and subcritical measurements performed in material balance areas of the Y-12 National Security Complex (NSC) and other facilities. Because of the high cost and large amount of facility time related to these measurements, it is very cost-effective to mine the existing data to produce NEA reactor physics benchmarks. The information for many of these measurements is documented in critical experiment and subcritical experiment logbooks at the International Criticality Safety Benchmark Program at Idaho National Laboratory and in Laboratory Records at Oak Ridge National laboratory.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

The Rodinia Benchmark Suite in SYCL

As opposed to the Open Computing Language (OpenCL) programming model in which host and device codes are generally written in different programming languages, SYCL can combine host and device codes for an application in a type-safe way to improve development productivity and performance portability. Rodinia is a widely used benchmark suite for heterogeneous computing. Hence, we port the OpenCL implementations of the benchmark suite to SYCL manually. The SYCL benchmark suite is an opensource project for tracking the development of the mainstream SYCL compilers, and for developers and researchers interested in programming productivity, performance analysis, and portability across different computing platforms. We organize the remainder of the report as follows. Section II introduces the SYCL programming model, shows the major differences between an OpenCL program and a SYCL program, and gives a summary of the benchmark suite. In Section III, we describe our SYCL implementations in more details. We evaluate the SYCL benchmarks on the Intel® central processing units (CPUs) and graphics processing units (GPUs) in Section IV. Section V concludes the report.

97 MATHEMATICS AND COMPUTING↗

(U) Updated Godiva-IV Benchmark Preview

A note of errata prepended to the Godiva-IV delayed-critical benchmark (HEU-METFAST-086) identifies two corrections that need to be made to the model: The glory hole in the spindle should be made larger, and the height of the safety block should be made smaller (and therefore its density made larger). In addition, the safety block at its full-in position is closer to the inner subassembly plate than was modeled in the benchmark. These changes have been made to HEU-METFAST-086 Case 4 in order to estimate the effect on k eff and on the neutron flux spectrum. Using smaller separation, a smaller safety block, and a larger glory hole caused k eff to increase by 453 ± 1 pcm from the benchmark. The latest nuclear data, ENDF-B/VIII.0, have also been used; this caused k eff to increase another 37 ± 1 pcm. Flux spectra were compared in a modeled fission foil and at three external point detectors. Within the fission foil, using ENDF/B-VIII.0 for Godiva-IV induced changes in the flux spectrum similar in size to the changes due to using smaller separation, a smaller safety block, and a larger glory hole. At the point detectors, using ENDF/B-VIII.0 induced changes in the flux spectrum much larger than those due to changing the model. In other words, the corrections to the benchmark model cause a large increase in k eff , but the changes to the neutron flux spectrum are small compared to those caused by using the latest nuclear data. This study presents a preview of results expected during the reevaluation of the Godiva IV benchmark, but it is not a substitute for the full reevaluation

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

The Rodinia Benchmarks in SYCL

The Rodinia Benchmark Suite in SYCL was first published as a technical report at Argonne National Laboratory in June 2020. While porting more programs to SYCL, the Rodinia benchmark suites were merged into the heterogenous computing benchmarks (HeCBench). Since then, the source codes and scripts of the benchmark suite have been updated to fix compile- and run-time issues that users encountered. With the recent SYCL compilers from Codeplay and Intel, this report presents the experimental results of evaluating these benchmarks on the Intel CPU and GPU devices.

97 MATHEMATICS AND COMPUTING↗