Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 325 records · Page 18

Benchmarking of the FENDL-3 Neutron Cross-section Data Starter Library for Fusion Applications

This paper summarizes the benchmark analyses performed in a joint effort of ENEA (Italy), JAEA (Japan), KIT (Germany), and the University of Wisconsin (USA) on a computational ITER benchmark and a series of 14 MeV neutron benchmark experiments. The computational benchmark revealed a modest increase of the neutron flux levels in the deep penetration regions and a substantial increase of the gas production in steel components. The comparison to experimental results showed good agreement with no substantial differences between FENDL-3.0 and FENDL-2.1 for most of the responses. In general, FENDL-3 shows an improved performance for fusion neutronics applications.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Spin Splitting Energy of Transition Metals: A New, More Affordable Wave Function Benchmark Method and Its Use to Test Density Functional Theory

Accurately predicting the spin splitting energy of chemical species is important for understanding their reactivity and magnetic properties, but it is very challenging, especially for molecules containing transition metals. One impediment to progress is the scarcity of accurate benchmark data. Here we report a set of calculations designed to yield reliable benchmarks for simple transition-metal complexes that can be used to test density functional methods that are affordable for large systems of more practical interest. Various wave function methods are tested against experiment for Fe 2+ , Fe 3+ , and Co 3+ , including CASSCF, CASPT2, CASPT3, MRCISD, MRCISD+Q, ACPF, AQCC, CCSD(T), and CASPT2/CCSD(T) and also a new method called CASPT2.5, which is performed by taking the average of the CASPT2 and CASPT3 energies. Furthermore, we find that MRCISD+Q, ACPF, and AQCC require smaller active spaces for good accuracy than are required by CASPT2 and CASPT3, and this aspect may be important for calculations on larger molecules; here we find that CASPT2.5 extrapolated to a complete basis set is the most suitable method—in terms of computational cost and in terms of accuracy on monatomic systems—and therefore we chose this method for molecular benchmarks. Then Kohn–Sham density functional calculations with 60 exchange-correlation functionals are tested for FeF 2 , FeCl 2 , and CoF 2 . We find that MN15-L, M06-SX, and revM06 have very good agreement with CASPT2.5 benchmarks in terms of both the spin splitting energy and the optimized geometry for each spin state. In addition, we recommend def2-TZVP as the most suitable basis set to perform density functional calculations for molecular spin splitting energies; extra polarization functions in the basis set do not help to increase the accuracy of the spin splitting energy in KS calculations.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Scientific machine learning benchmarks

Deep learning has transformed the use of machine learning technologies for the analysis of large experimental datasets. In science, such datasets are typically generated by large-scale experimental facilities, and machine learning focuses on the identification of patterns, trends and anomalies to extract meaningful scientific insights from the data. In upcoming experimental facilities, such as the Extreme Photonics Application Centre (EPAC) in the UK or the international Square Kilometre Array (SKA), the rate of data generation and the scale of data volumes will increasingly require the use of more automated data analysis. Furthermore, at present, identifying the most appropriate machine learning algorithm for the analysis of any given scientific dataset is a challenge due to the potential applicability of many different machine learning frameworks, computer architectures and machine learning models. Historically, for modelling and simulation on high-performance computing systems, these issues have been addressed through benchmarking computer applications, algorithms and architectures. Extending such a benchmarking approach and identifying metrics for the application of machine learning methods to open, curated scientific datasets is a new challenge for both scientists and computer scientists. Here, we introduce the concept of machine learning benchmarks for science and review existing approaches. As an example, we describe the SciMLBench suite of scientific machine learning benchmarks.

42 ENGINEERING↗

A critical comparison of CH∙∙∙π versus π∙∙∙π interactions in the benzene dimer: obtaining benchmarks at the CCSD(T) level and assessing the accuracy of lower scaling methods

In this study, we have established CCSD(T)/CBS (Complete Basis Set) limits for 3 stationary points on the benzene dimer potential energy surface, corresponding to the π∙∙∙π (parallel displaced or PD(C 2h ), minimum) and CH∙∙∙π (T-shaped or T(C 2v ), transition state) and tilted T-shaped (or TT(Cs), minimum) bonding scenarios considering both the structure and binding energy. The CCSD(T)/CBS binding energies are -2.65 ± 0.02 (PD), -2.74 ± 0.03 (T), and -2.83 ± 0.01 kcal mol -1 (TT). To this end, the CH∙∙∙π is ~0.2 kcal mol -1 stronger than the π∙∙∙π interaction, whereas the tilting of the CH donating benzene molecule with respect to the other benzene is worth 0.1 kcal mol -1 . As previously discussed in the literature, the MP2 level of theory does not provide a close match for either the energy or structure, yet the SCS-MP2 yields structures in excellent agreement with respect to the CCSD(T) result. It is found that the SCS-MI-MP2 also gives optimized structures very close to SCS-MP2 (within ~0.01 Å of the benchmark). Despite the closer match in structure, the spin-biased MP2 methods (SCS-, SCS-MI-, and SOS-MP2) incorrectly predict the relative stabilities of the isomers. That said, none of the spin biased MP2 methods offers a good compromise between energy and structure for the systems examined. Finally, the CCSD(T)/CBS benchmarks were used to assess the performance of 13 DFT functionals selected from different rungs of Jacob's ladder. Several functionals such as TPSS-D3, B3LYP-D3, B97-D, B97-D3, and B2PLYP-D3 provided a good description of the binding energies for both CH∙∙∙π and π∙∙∙π interactions, yielding values within 6% of the CCSD(T)/CBS benchmark values. Unlike the MP2 methods, these functionals correctly predict the relative stability of the PD(C 2h ) and T(C 2v ) dimers. Further, we find that there is no systematic improvement as Jacob's ladder is ascended (increased complexity of functional). The best functionals that result in a good compromise between structure and energy accuracy are B97-D3 and B2PLYP-D3 for both the CH∙∙∙π and π∙∙∙π interaction. Despite the impressive performance of these functionals, a challenge that remains is ensuring the transferability of these density functionals in accurately describing the interaction between dimers of larger aromatic molecules, the latter requiring high-level benchmarks for these systems.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Availability of Critical Benchmark Experiments for the Pebble Tanker Transportation Model for Nuclear Criticality Safety Validation of TRISO Pebbles

This study addresses the need for comprehensive investigations into TRi-structural ISOtropic (TRISO) fuel pebble transportation validation. In this work, an exploratory model, the pebble tanker(PT), was developed with the aim of facilitating the validation of nuclear criticality safety calculations in the context of industrial-scale transportation of TRISO fuel. The PT model was designed to investigate the availability and applicability of critical benchmark experiments crucial for assessing the transportation of these pebbles. This work incorporated sensitivity/uncertainty (S/U) similarity studies to quantify the applicability of critical benchmark experiments and to address nuclear data uncertainties in the context of TRISO transportation. Two container models were investigated: one for the Hermes-type pebble and one for the Pebble Bed Modular Reactor (PBMR)–type pebble. The models were simplified, considering fuel, containment, and either water or air, to enable a focus on the underlying physics of applications involving TRISO fuel pebbles using the PT model. A crucial aspect under consideration was the capacity of the transport package to hold pebbles while ensuring subcriticality in the flooded state. An approach in the criticality validation process involves assessing the similarity between systems through an integral index parameter evaluation. This involves calculating a correlation coefficient (referred to as c k ) based on shared nuclear data–induced uncertainty between a benchmark experiment and the application of the PT model. To facilitate this analysis, the SCALE tools, particularly the CSAS6-Shift, TSUNAMI-3D-Shift, and TSUNAMI-IP sequences, were employed for comprehensive studies in neutronics and S/U analysis. Our findings showed that there are sufficient critical experimental benchmarks to perform this validation of the PT model in the most reactive state, i.e. when the tanker is flooded. This paper provides valuable insights into validating a transport package for Generation IV TRISO fuel pebbles.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Assessment of Critical Experiment Benchmark Applicability to a Large-Capacity HALEU Transportation Package Concept

This work presents an assessment of the applicability of existing benchmark critical experiments to the criticality safety code validation for a large-capacity high-assay low-enriched uranium (HALEU) transportation package concept. Numerous next-generation nuclear reactor designs require HALEU fuel, which is characterized by an enrichment between 5 and 20 wt% 235 U. The U.S. Department of Energy (DOE) has proposed to recover and downblend highly enriched uranium from DOE-owned used nuclear fuel to accelerate the demonstration of commercially viable microreactor technologies. One element of the infrastructure needed to demonstrate HALEU-fueled reactors is the ability to safely transport enriched product to be used for fuel fabrication. There is uncertainty as to whether existing critical benchmark experiment data are sufficient to support criticality safety code validation for HALEU transportation applications. The anticipated chemical form of the HALEU in the proposed transportation concept is UO 2 with 20 wt% 235 U/U. The concept uses a combination of an existing transportation packaging design and a novel basket design, including borated aluminum flux traps. The basket provides space for 18 reusable, stainless steel canisters that contain the HALEU. In 10 CFR 71, normal conditions of transport (NCTs) and hypothetical accident conditions (HACs) are defined for fissile material transportation packages. NCT and HAC KENO-VI models of the transportation package were developed using the Standardized Computer Analyses for Licensing Evaluation (SCALE) 6.2.3 computer code package, and optimum moderation conditions were determined using the SCALE SAMPLER sequence. The SCALE Tools for Sensitivity and Uncertainty Analysis Methodology Implementation (TSUNAMI) sequences were then used to compare the neutronic characteristics of 1584 International Criticality Safety Benchmark Evaluation Project benchmark critical experiments with the NCT and HAC HALEU transportation models. The TSUNAMI integral correlation coefficient c k was the criterion used to rank neutronic similarity. Thirty-four experiments were identified as similar (c k ≥ 0.9) to the NCT model, and 55 experiments were identified as similar to the HAC model. Hundreds of experiments were also identified as at least marginally similar (c k ≥ 0.8) to both models. The results indicate that additional critical experiments are unlikely to be needed to support HALEU transportation criticality safety analyses for package concepts similar to the concept package analyzed.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Code Benchmark of the HTTF Pressurized Conduction Cooldown Test Using SAM

The High Temperature Test Facility (HTTF) at Oregon State University is an integral system test facility to simulate postulated reactor transients of prismatic high-temperature gas-cooled reactors(HTGRs). A series of test campaigns was launched, providing abundant test data that could be used to benchmark reactor system analysis codes like the System Analysis Module (SAM). In this study, a SAM model of the facility is developed based on the two-dimensional (2D) ring model approach. All components including the ceramic matrix, graphite heaters, helium coolant channels, core barrel, upcomer, pressure vessel, and reactor cavity cooling system are modeled as concentric cylindrical rings. The model is used to simulate one of the benchmark problems-Pressurized Conduction Cooldown (PCC)-within the scope of the Organisation for Economic Co-operation and Development Nuclear Energy Agency International HTTF Benchmark. The simulations consist of two parts. In the first part, operating and boundary conditions as well as thermophysical properties of materials are specified for the benchmark problem. In this work, results from the first part will be used in code-to-code comparison. In the second part, the SAM model is used to simulate Test PG-27, which is the first PCC test carried out in the HTTF, with only two of the ten heater banks activated. The results in the second part are used for code-to-data comparison. Because the helium coolant flow rate is not measured in this facility, it is estimated using the input power and inlet/outlet coolant temperatures. Additionally, radial heat flow in the ceramic blocks is complicated by hundreds of cylindrical coolant channels and heater rods embedded in them. As such, it is necessary to deduce an effective thermal conductivity for the ceramic to analyze the core thermal behavior. SAM predictions of the helium coolant and ceramic temperatures are compared with test data measured in three equivalent sectors. Overall, the SAM results agree reasonably well with test data within the variation of data among the three sectors, which demonstrates SAM's capability in capturing transient effects in HTGR using the simplified 2D ring model.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Multicode benchmark on simulated Ti K-edge x-ray absorption spectra of Ti-O compounds

X-ray absorption spectroscopy (XAS) is an element-specific materials characterization technique that is sensitive to structural and electronic properties. First-principles simulated XAS has been widely used as a powerful tool to interpret experimental spectra and draw physical insights. Recently, there has also been growing interest in building computational XAS databases to enable data analytics and machine learning applications. However, there are non-trivial differences among commonly used XAS simulation codes, both in underlying theoretical formalism and in technical implementation. Reliable and reproducible computational XAS databases require systematic benchmark studies. In this work, we benchmarked Ti K-edge XAS simulations of ten representative Ti-O binary compounds, which we refer to as the Ti-O-10 dataset, using three state-of-the-art codes: XSPECTRA, OCEAN and exciting. We systematically studied the convergence behavior with respect to the input parameters and developed a workflow to automate and standardize the calculations to ensure converged spectra. Our benchmark comparison considers a 35 eV spectral range starting from the K-edge onset, representative of widely used near-edge spectra. Quantitative comparison over this range is based on Spearman’s rank correlation score (r sp ). Our results show: (1) the two Bethe-Salpeter equation (BSE) codes (OCEAN and exciting) have excellent agreement with an average $r$ $sp$ of 0.998; (2) good agreement is obtained between the core-hole potential code (XSPECTRA) and BSE codes (OCEAN and exciting) with an average $r$ $sp$ of 0.990, and this smaller $r$ $sp$ reflects the noticeable differences in the main edge spectral shape that can be primarily attributed to the difference in the strength of the screened core-hole potential; (3) simulations from both methods overall reproduce well the main experimental spectral features of rutile and anatase, and the different treatments of the screened core-hole potential have visible impact on pre-edge intensities and the peak ratio of the main edge; (4) there exist moderate differences in the relative edge alignment of the three codes with a standard deviation of about 0.2 eV, which arise from multiple contributions including the frozen core approximation, final state effects, and different approximations used for the self-energy correction. In conclusion, our benchmark study provides important standards for first-principles XAS simulations with broad impact in data-driven XAS analysis.

36 MATERIALS SCIENCE↗

Benchmark Generation and Simulation at Extreme Scale

The path to extreme scale high-performance computing (HPC) poses several challenges related to power, performance, resilience, productivity, programmability, data movement, and data management. Investigating the performance of parallel applications at scale on future architectures and the performance impact of different architectural choices is an important component of HPC hardware/software co-design. Simulations using models of future HPC systems and communication traces from applications running on existing HPC systems can offer an insight into the performance of future architectures. This work targets technology developed for scalable application tracing of communication events. It focuses on extreme-scale simulation of HPC applications and their communication behavior via lightweight parallel discrete event simulation for performance estimation and evaluation. Instead of simply replaying a trace within a simulator, this work promotes the generation of a benchmark from traces. This benchmark is subsequently exposed to simulation using models to reflect the performance characteristics of future-generation HPC systems. This technique provides a number of benefits, such as eliminating the data intensive trace replay and enabling simulations at different scales. The presented work features novel software co-design aspects, combining the ScalaTrace tool to generate scalable trace files, the ScalaBenchGen tool to generate the benchmark, and the xSim tool to assess the benchmark characteristics within a simulator.

Lagadapati, Mahesh↗

SRBench++: Principled Benchmarking of Symbolic Regression With Domain-Expert Interpretation

Symbolic regression searches for analytic expressions that accurately describe studied phenomena. The main promise of this approach is that it may return an interpretable model that can be insightful to users, while maintaining high accuracy. The current standard for benchmarking these algorithms is SRBench, which evaluates methods on hundreds of datasets that are a mix of real-world and simulated processes spanning multiple domains. At present, the ability of SRBench to evaluate interpretability is limited to measuring the size of expressions on real-world data, and the exactness of model forms on synthetic data. In practice, model size is only one of many factors used by subject experts to determine how interpretable a model truly is. Furthermore, SRBench does not characterize algorithm performance on specific, challenging sub-tasks of regression such as feature selection and evasion of local minima. In this work, we propose and evaluate an approach to benchmarking SR algorithms that addresses these limitations of SRBench by 1) incorporating expert evaluations of interpretability on a domain-specific task, and 2) evaluating algorithms over distinct properties of data science tasks. We evaluate 12 modern symbolic regression algorithms on these benchmarks and present an in-depth analysis of the results, discuss current challenges of symbolic regression algorithms and highlight possible improvements for the benchmark itself.

97 MATHEMATICS AND COMPUTING↗

Using benchmarks for radiation testing of microprocessors and FPGAs

Performance benchmarks have been used over the years to compare different systems. These benchmarks can be useful for researchers trying to determine how changes to the technology, architecture, or compiler affect the system's performance. No such standard exists for systems deployed into high radiation environments, making it difficult to assess whether changes in the fabrication process, circuitry, architecture, or software affect reliability or radiation sensitivity. In this paper, we propose a benchmark suite for high-reliability systems that is designed for field-programmable gate arrays and microprocessors. As a result, we describe the development process and report neutron test data for the hardware and software benchmarks.

97 MATHEMATICS AND COMPUTING↗

Benchmarking black hole heat engines, II

We extend to nonstatic black holes our benchmarking scheme that allows for cross–comparison of the efficiencies of asymptotically AdS black holes used as working substances in heat engines. We use a circular cycle in the [Formula: see text] plane as the benchmark cycle. We study Kerr black holes in four spacetime dimensions as an example. As in the static case, we find an exact formula for the benchmark efficiency in an ideal gas-like limit, which may serve as an upper bound for rotating black hole heat engines in a thermodynamic ensemble with fixed angular velocity. We use the benchmarking scheme to compare Kerr to static black holes charged under Maxwell and Born–Infeld sectors.

Astronomy & Astrophysics↗

HamPerf: A Hamiltonian-Oriented Approach to Quantum Benchmarking

Quantum computing technologies are undergoing rapid development. The different qubit modalities being considered for quantum computing each have their strengths and weaknesses, making it challenging to compare their performance relative to each other and the state-of-the-art in classical high-performance computing. To better understand the utility of a given quantum processor and to assess when and how it will be able to advance the frontiers of computational science, researchers need a robust approach to quantum benchmarking. A variety of approaches have been proposed, many of which characterize the presence of noise in current quantum devices. These efforts include component-level performance metrics, such as randomized benchmarking and gate set tomography; high-level application-dependent metrics; and devicelevel metrics, such as the Quantum Volume. However, it remains unclear how low-level metrics, such as fidelities and decoherence times, and global device metrics, such as Quantum Volume, relate to the computational utility and practical limitations of quantum processors to solve useful problems. In this paper, we describe our Hamiltonian-oriented approach to quantum benchmarking called HamPerf. Where previous application-dependent approaches specify a suite of benchmarking circuits inspired by applications, we place the problem Hamiltonian at the center. Our strategy allows us to probe the computational performance of a quantum processor on standardized and relevant problem sets, agnostic of the algorithms and hardware used to solve them; it also provides fundamental insights into how device characteristics correlate with computational utility.

Butko, Anastasiia↗

Update to the Microcontroller Benchmark for Radiation Testing

LANL developed a benchmark of software code for radiation testing of microprocessors several years ago, and it was published under an open-source license on GitHub. Publishing the software is necessary for other researchers to adopt and implement this benchmark for radiation testing of other microprocessors to standardize test practices so that test data can be compared across different microprocessors. The original codes have been used several times by other organizations to test a wide range of microcontrollers and microprocessors. After several years of research, LANL is ready to update the benchmark. Changes include: 1. Addition of new codes that allow common software codes to be tested, 2. Addition of new codes that instrument more microprocessor circuitry, 3. Addition of input patterns that allow for a more compressive understanding of how the memory layout affects the sensitivity to radiation-induced faults and better use of automated test pattern generation standards, and 4. Modification of current codes for faster and more resilient detection, reporting and correction of radiation-induced faults. These codes have been tested by LANL researchers over the last few years, which has been published in the open literature. As the code base for the new benchmarks are stable, it is time to release the update to the GitHub repository, where the original codes were released.

Quinn, Heather↗

SEPATH: benchmarking the search for pathogens in human tissue whole genome sequence data leads to template pipelines

Background: Human tissue is increasingly being whole genome sequenced as we transition into an era of genomic medicine. With this arises the potential to detect sequences originating from microorganisms, including pathogens amid the plethora of human sequencing reads. In cancer research, the tumorigenic ability of pathogens is being recognized, for example, Helicobacter pylori and human papillomavirus in the cases of gastric non-cardia and cervical carcinomas, respectively. As of yet, no benchmark has been carried out on the performance of computational approaches for bacterial and viral detection within host-dominated sequence data. Results: We present the results of benchmarking over 70 distinct combinations of tools and parameters on 100 simulated cancer datasets spiked with realistic proportions of bacteria. mOTUs2 and Kraken are the highest performing individual tools achieving median genus-level F1 scores of 0.90 and 0.91, respectively. mOTUs2 demonstrates a high performance in estimating bacterial proportions. Employing Kraken on unassembled sequencing reads produces a good but variable performance depending on post-classification filtering parameters. These approaches are investigated on a selection of cervical and gastric cancer whole genome sequences where Alphapapillomavirus and Helicobacter are detected in addition to a variety of other interesting genera. Conclusions: We provide the top-performing pipelines from this benchmark in a unifying tool called SEPATH, which is amenable to high throughput sequencing studies across a range of high-performance computing clusters. SEPATH provides a benchmarked and convenient approach to detect pathogens in tissue sequence data helping to determine the relationship between metagenomics and disease.

Viral↗

Status of the KRUSTY Benchmark Modeling and Uncertainty Analysis

The U.S. National Aeronautics and Space Administration’s (NASA) desire for a small nuclear reactor for space applications led to a prototype assembly, Kilopower Reactor Using Stirling Technology (KRUSTY). The KRUSTY experiments included four phases: component critical, cold critical, warm critical, and high temperature. The KRUSTY benchmark described in this paper includes five configurations from the component critical experiments that were performed at the National Criticality Experiments Research Center (NCERC) beginning in November 2017. Preliminary results have been reported. The benchmark is now nearing completion (although it is still preliminary until formally accepted by the International Criticality Safety Benchmark Evaluation Project [ICSBEP] working group). In this paper, we present the updated benchmark and its uncertainties. These results were computed using MCNP6.2 ® with ENDF/B-VIII.0 cross sections.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Benchmark Imagery for Assessing Geospatial Semantic Content Extraction Algorithms (Final Report)

Benchmark Imagery for Assessing Geospatial Semantic Content Extraction Algorithms (the “Benchmark Imagery Project”) was a three-year endeavor funded by the Enabling Capabilities (EC) Team of DOE/NNSA’s Office of Proliferation Detection R&D, NA-22, and carried out through a collaborative effort between Los Alamos National Laboratory (LANL) and the Lawrence Livermore National Laboratory (LLNL). The primary objective was to create a set of benchmark imagery for validating algorithms designed to extract semantic content from geospatial images of industrial facilities. The goal was not to develop algorithms; rather, it was to provide information that could be used to do so. The secondary objective was to demonstrate how this information could be used for algorithm validation. This report summarizes the results of the Benchmark Imagery Project.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Clean Energy Manufacturing Analysis Center Benchmark Report: Framework and Methodologies

This report documents the CEMAC methodologies for developing and reporting annual global clean energy manufacturing benchmarks. The report reviews previously published manufacturing benchmark reports and foundational data, establishes a framework for benchmarking clean energy technologies, describes the CEMAC benchmark analysis methodologies, and describes the application of the methodologies to the manufacturing of four specific clean energy technologies.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗