Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11

Status of the International Criticality Safety Benchmark Evaluation Project

The International Criticality Safety Benchmark Evaluation Project (ICSBEP) has continued its work generating evaluations of new and historical benchmark experiments since the last update to the nuclear criticality safety (NCS) community at the 12th International Conference on Nuclear Criticality Conference held in 2023. One additional version of the ICSBEP Handbook has been published since that update, and the Technical Review Group (TRG) held two in-person meetings to review and approve additional benchmarks. The 2022 and 2023 editions of the handbook were combined into one release (published in November 2024) and contained 13 new evaluations with 46 different configurations and two major revisions to existing evaluations. The 2024 version of the handbook, currently under publication review, will contain two new evaluations with 15 new configurations and one major revision to HEU-MET-FAST-028, the evaluation of Flattop with a uranium core. The ICSBEP TRG met again in person in April 2025 to review benchmarks for the 2025 ICSBEP Handbook and final comment resolution is currently ongoing. Many of the new benchmarks represent contemporaneous experiments that have been specifically optimized to provide validation cases relevant to the NCS community. One major area of focus for new critical experiments is to target the sparsely populated intermediate energy (or resonance) region. Another focus of many of the new benchmarks is to provide experiments sensitive to different materials, such as chlorine, hafnium, tantalum, titanium, molybdenum, chromium, and polymethyl methacrylate (PMMA, or Lucite). The ICSBEP continues to deliver high-quality, peer reviewed evaluations of integral experiments relevant to the nuclear data community.

HEU-MET-FAST-028↗

Intern-Artificial Intelligence Benchmarking

Benchmarks provide a standardized method for evaluating different AI models, enabling reproducibility and comparison between models, and facilitating scientific progress. As AI models continue to develop rapidly, incorporating new datasets, capabilities, and architectures becomes more complicated. Therefore, the current static benchmarks become increasingly irrelevant. The MLCommons team argues that to make AI benchmarks more relevant, it involves making the benchmarks themselves more dynamic, as well as technical innovations that make it easier for scientists and researchers at all levels to use and contribute to the benchmarks. The current progress in technical innovation is a software that allows for a detailed view of a collection of AI benchmarks to be output in various formats that are easily readable and accessible.

Krishnan, Anjay [Fermilab]↗

Classifying and rating AI benchmarks

We created a set of standards to efficiently evaluate AI benchmarks through objective means. Although prevalent, especially in recent times, AI benchmarks have no single way to measure their effectiveness. The MLCommons team provided a set of criteria for evaluating benchmarks, although the criteria lacks a clearly defined set of evaluation rules. We created a rubric with preset factors to efficiently and objectively evaluate a benchmark’s quality. We created a software framework for processing lists of benchmarks for visualization. The framework and rating system allows researchers to quickly check if their benchmarks are effective.

Shiraishi, Reece [Fermilab]↗

Status of the International Criticality Safety Benchmark Evaluation Project

The International Criticality Safety Benchmark Evaluation Project (ICSBEP) has continued its work generating evaluations of new and historical criticality benchmark experiments since the last update to the nuclear criticality safety community at the 11th International Conference on Nuclear Criticality Safety (ICNC 2019) in Paris, France. Three additional versions of the ICSBEP Handbook have been published since that update, and the Technical Review Group (TRG) held two in-person (in 2019 and 2023) and three virtual (2020 and 2021) meetings to review and approve additional benchmarks. The 2019 edition of the ICSBEP Handbook included five new evaluations with 79 new configurations, the 2020 version of the ICSBEP Handbook contained five new evaluations totaling 76 new configurations, and the 2021 version of the ICSBEP handbook contained five new evaluations with a total of 57 different configurations. The ICSBEP TRG met in October and December 2021, to review benchmarks for potential inclusion in the 2022 ICSBEP Handbook, with seven evaluations receiving provisional approval pending resolution of review group comments. Final comment resolution for some of these evaluations is currently underway and handbook publication should be completed soon. The ICSBEP TRG met again in person in April 2023 to review benchmarks for the 2023 ICSBEP Handbook, provisionally approving 7 new evaluations. The ICSBEP continues to deliver high-quality, peer reviewed evaluations of experiments relevant to the nuclear criticality safety community.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Autotuning PolyBench benchmarks with LLVM Clang/Polly loop optimization pragmas using Bayesian optimization

Here, we develop a ytopt autotuning framework that leverages Bayesian optimization to explore the parameter space search and compare four different supervised learning methods within Bayesian optimization and evaluate their effectiveness. We select six of the most complex PolyBench benchmarks and apply the newly developed LLVM Clang/Polly loop optimization pragmas to the benchmarks to optimize them. We then use the autotuning framework to optimize the pragma parameters to improve their performance. The experimental results show that our autotuning approach outperforms the other compiling methods to provide the smallest execution time for the benchmarks syr2k, 3mm, heat-3d, lu, and covariance with two large datasets in 200 code evaluations for effectively searching the parameter spaces with up to 170,368 different configurations. We find that the Floyd-Warshall benchmark did not benefit from autotuning. To cope with this issue, we provide some compiler option solutions to improve the performance. Then we present loop autotuning without a user's knowledge using a simple mctree autotuning framework to further improve the performance of the Floyd-Warshall benchmark. We also extend the ytopt autotuning framework to tune a deep learning application.

79 ASTRONOMY AND ASTROPHYSICS↗

Benchmarking Initiatives

Benchmarking has been recognized as an efficient tool for model validation as well as scientific collaboration. Several workshop series have been established to foster the benchmarking idea: setting up test cases with increasing complexity for method development and code comparison. In addition to representing model complexity, one of the key efforts is to develop codes that are suitable for modern HPC platforms such as PetaFlop supercomputers. It has been realized that those challenges are beyond single team capabilities. The Subsurface Environmental Simulation Benchmarking (SSBench) working group was initiated to develop, publish, and make publicly available well-described benchmark problems that could be used to demonstrate simulator conformance with norms established by the subsurface science and engineering community. SSBench supports the development of environmentally-relevant benchmark problem sets that provide rigorous tests for process model conceptualization, numerical implementation, process coupling, and accuracy. The initial SSBench focus on multicomponent reactive transport modeling reflects the expanding range of mechanisms and increasing complexity needed to address issues relevant to climate change, nuclear waste management, contaminant remediation, and pollution prevention.

benchmarking↗

Defining and applying an electricity demand flexibility benchmarking metrics framework for grid-interactive efficient commercial buildings

Building demand flexibility (DF) research has recently gained attention. To unlock building DF as a predictable grid resource, we must establish a quantitative understanding of the resource size, performance variability, and predictability based on large empirical datasets. Researchers have proposed various sets of theoretical metrics to measure this performance. Some metrics have been applied to simulation results, but most fall short of exploring the complexities in real building applications. There are practical metrics used in individual demand response field studies but they alone cannot fulfil the job of DF benchmarking across a diverse group of buildings. The electrical grid's geographically diverse and changing nature presents challenges to comparing building DF performance measured under different conditions (i.e., benchmarking DF). To address this challenge, a novel DF benchmarking framework focused on load shedding and shifting is presented; the foundation is a set of simple, proven single-event metrics with attributes describing event conditions. These enable benchmarking and visualization in different dimensions for identifying trends that represent how these attributes influence DF. To test its feasibility and scalability, the DF framework was applied to two case studies of 11 office buildings and 121 big-box retail buildings with demand response participation data. Furthermore, these examples provided a pathway for using both building level benchmarking and aggregation to extract insights into building DF about magnitude, consistency, and influential factors. Potential applications of the framework and real-world values have been identified for grid and building stakeholders.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Benchmark Credentialing Results for NRG-BR001: The First National Cancer Institute-Sponsored Trial of Stereotactic Body Radiation Therapy for Multiple Metastases

Purpose: The NRG-BR001 trial is the first National Cancer Institute–sponsored trial to treat multiple (range 2-4) extracranial metastases with stereotactic body radiation therapy. Benchmark credentialing is required to ensure adherence to this complex protocol, in particular, for metastases in close proximity. The present report summarizes the dosimetric results and approval rates. Methods and Materials: The benchmark used anonymized data from a patient with bilateral adrenal metastases, separated by <5 cm of normal tissue. Because the planning target volume (PTV) overlaps with organs at risk (OARs), institutions must use the planning priority guidelines to balance PTV coverage (45 Gy in 3 fractions) against OAR sparing. Submitted plans were processed by the Imaging and Radiation Oncology Core and assessed by the protocol co-chairs by comparing the doses to targets, OARs, and conformity metrics using nonparametric tests. Results: Of 63 benchmarks submitted through October 2015, 94% were approved, with 51% approved at the first attempt. Most used volumetric arc therapy (VMAT) (78%), a single plan for both PTVs (90%), and prioritized the PTV over the stomach (75%). The median dose to 95% of the volume was 44.8 ± 1.0 Gy and 44.9 ± 1.0 Gy for the right and left PTV, respectively. The median dose to 0.03 cm{sup 3} was 14.2 ± 2.2 Gy to the spinal cord and 46.5 ± 3.1 Gy to the stomach. Plans that spared the stomach significantly reduced the dose to the left PTV and stomach. Conformity metrics were significantly better for single plans that simultaneously treated both PTVs with VMAT, intensity modulated radiation therapy, or 3-dimensional conformal radiation therapy compared with separate plans. No significant differences existed in the dose at 2 cm from the PTVs. Conclusions: Although most plans used VMAT, the range of conformity and dose falloff was large. The decision to prioritize either OARs or PTV coverage varied considerably, suggesting that the toxicity outcomes in the trial could be affected. Several benchmarks met the dose-volume histogram metrics but produced unacceptable plans owing to low conformity. Dissemination of a frequently-asked-questions document improved the approval rate at the first attempt. Benchmark credentialing was found to be a valuable tool for educating institutions about the protocol requirements.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

The PAC-MAN model: Benchmark case for linear acoustics in computational physics

Benchmark cases in the field of computational physics, on the one hand, have to contain a certain complexity to test numerical edge cases and, on the other hand, require the existence of an analytical solution, because an analytical solution allows the exact quantification of the accuracy of a numerical simulation method. This dilemma causes a need for analytical sound field formulations of complex acoustic problems. A well known example for such a benchmark case for harmonic linear acoustics is the “Cat's Eye model”, which describes the three-dimensional sound field radiated from a sphere with a missing octant analytically. In this paper, a benchmark case for two-dimensional (2D) harmonic linear acoustic problems, viz., the “PAC-MAN model”, is proposed. The PAC-MAN model describes the radiated and scattered sound field around an infinitely long cylinder with a cut out sector of variable angular width. While the analytical calculation of the 2D sound field allows different angular cut-out widths and arbitrarily positioned line sources, the computational cost associated with the solution of this problem is similar to a 1D problem because of a modal formulation of the sound field in the PAC-MAN model. - Highlights: • A benchmark case for 2D harmonic linear acoustic problems is proposed. • The PAC-MAN geometry is used as emitter/scatterer in this benchmark. • An analytic formulation of radiated and scattered sound is derived. • Line and disk sources, plane waves, and surface vibrations are considered.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Benchmarking of HEU Mental Annuli Critical Assemblies with Internally Reflected Graphite Cylinder

Three experimental configurations of critical assemblies, performed in 1963 at the Oak Ridge Critical Experiment Facility, which are assembled using three different diameter HEU annuli (15-9 inches, 15-7 inches and 13-7 inches) metal annuli with internally reflected graphite cylinder are evaluated and benchmarked. The experimental uncertainties which are 0.00055, 0.00055 and 0.00055 respectively, and biases to the detailed benchmark models which are -0.00179, -0.00189 and -0.00114 respectively, were determined, and the experimental benchmark keff results were obtained for both detailed and simplified model. The calculation results for both detailed and simplified models using MCNP6-1.0 and ENDF VII.1 agree well to the benchmark experimental results with a difference of less than 0.2%. These are acceptable benchmark experiments for inclusion in the ICSBEP Handbook.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Benchmark of the Kilowatt Reactor Using Stirling TechnologY (KRUSTY) Component Critical Configurations

Kilowatt Reactor Using Stirling TechnologY (KRUSTY) was a prototype for the U.S. National Aeronautics and Space Administration’s Kilopower Program. KRUSTY has a highly enriched uranium–molybdenum alloy (with 7.65 wt% molybdenum) annular core reflected by beryllium oxide with an outer stainless steel shield. Five configurations from the experimental campaign were chosen to be evaluated as benchmark cases. Uncertainties were evaluated in five categories: (1) criticality measurement, (2) mass and density, (3) dimensions, (4) material compositions, and (5) positioning. The largest contribution to the overall uncertainty in each case was from the radial alignment of the movable platen. A simplified model was created to increase computational efficiency, and an average bias of –16 pcm was calculated due to the simplifications. Sample calculations were completed for each case using MCNP6.2, COG, and MC21, all with ENDF/B-VIII.0 nuclear data. For MCNP6.2, the average difference (absolute value) between the calculated and experimental $k_{eff}$ for the five configurations was 14 pcm for both the detailed and the simplified models. The $k_{eff}$ results from all three codes are within 1σ of the benchmark values. KRUSTY’s value as a benchmark is due to its sensitivity to beryllium and molybdenum. For beryllium, KRUSTY adds an 18th benchmark with a total cross-section sensitivity greater than 0.05%/%/(unit lethargy). For molybdenum, KRUSTY adds a 9th benchmark with a total cross-section sensitivity greater than 0.004%/%/(unit lethargy).

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

The MUSIC Critical Benchmark and Nuclear Data

The Measurement of Uranium Subcritical and Critical (MUSIC) experiment was a series of measurements of critical and subcritical configurations of bare highly enriched uranium. The goal was to compare measurement methods, analysis techniques, and simulation methods across regimes of criticality and to provide high-quality validation of 235 U nuclear data. A benchmark evaluation of the two critical configurations of the MUSIC experiments will soon be published in the release of the International Criticality Safety Benchmark Evaluation Project Handbook. The recent execution of the experiment aids in proper quantification of model simplifications and all uncertainties associated with the experiment. Historical benchmark evaluations are heavily relied on for uranium nuclear data validation despite the fact that the same level of documentation and comparable uncertainty analysis may not be present. The MUSIC evaluation is less likely to include “unknown unknowns” that could impede accurately modeling the system. Presented are both highly detailed and very simplified models, which represent the experimental configurations accurately, aiding the users of the benchmark for nuclear data or transport code validation. The sensitivities of k eff to nuclear data and nuclear data–related uncertainties are very similar between this experiment and previous bare uranium sphere experiments. In addition, the nuclear data uncertainties to any nuclides other than 235 U are small. For all these reasons, the recently evaluated MUSIC benchmark critical configurations could prove very useful for 235 U nuclear data validation. Currently, major libraries have good agreement with the experimental results, within 200 pcm for all nuclear data libraries, and within one standard deviation of the experimental result for most. Suggested nuclear data adjustments based on MUSIC and Lady Godiva are also presented, with posterior improvements to both the agreement in k eff and the uncertainty associated with the nuclear data.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

An Analytic Benchmark for Neutron Boltzmann Transport with Downscattering—Part IV: PFNS and $\bar{ν}$ Uncertainty Propagation

An analytic benchmark with continuous-energy cross sections was previously derived to validate criticality calculations. Here, to extend the utility of the analytic benchmark to verify the implementation of $\bar{ν}$ and prompt fission neutron spectrum (PFNS) uncertainty propagation methods, new simplified forms that are dependent on the incident (fission-causing) neutron energy, as well as the outgoing neutron energy for the PFNS, are introduced in this work. The analytical forms for the flux and adjoint flux are derived for the extended benchmark and used to determine the 𝑘-eigenvalue sensitivity to $\bar{ν}$ and PFNS. The 𝑘-eigenvalue uncertainty due to $\bar{ν}$ and PFNS is calculated for the analytic benchmark using simplified$\bar{ν}$ and PFNS representations based on the ENDF-B/VIII.0 239 Pu evaluation. Because of the low sensitivity of the analytic benchmark to the physical PFNS, a nonphysical high-sensitivity PFNS is also presented.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Random circuit block-encoded matrix and a proposal of quantum LINPACK benchmark

The LINPACK benchmark reports the performance of a computer for solving a system of linear equations with dense random matrices. Although this task was not designed with a real application directly in mind, the LINPACK benchmark has been used to define the list of TOP500 supercomputers since the debut of the list in 1993. We propose that a similar benchmark, called the quantum LINPACK benchmark, could be used to measure the whole machine performance of quantum computers. The success of the quantum LINPACK benchmark should be viewed as the minimal requirement for a quantum computer to perform a useful task of solving linear algebra problems, such as linear systems of equations. We propose an input model called the Random Circuit Block-Encoded Matrix (RACBEM), which is a proper generalization of a dense random matrix in the quantum setting. The RACBEM model is efficient to be implemented on a quantum computer and can be designed to optimally adapt to any given quantum architecture, with relying on a black-box quantum compiler. Besides solving linear systems, the RACBEM model can be used to perform a variety of linear algebra tasks relevant to many physical applications, such as computing spectral measures, time series generated by a Hamiltonian simulation, and thermal averages of the energy. We implement these linear algebra operations on IBM Q quantum devices as well as quantum virtual machines, and demonstrate their performance in solving scientific computing problems.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Applications of Integral Benchmark Data

The International Reactor Physics Experiment Evaluation Project (IRPhEP) and the International Criticality Safety Benchmark Evaluation Project (ICSBEP) provide evaluated integral benchmark data that may be used for validation of reactor physics / nuclear criticality safety analytical methods and data, nuclear data testing, advanced modeling and simulation, and safety analysis licensing activities. The handbooks produced by these programs are used in over 30 countries. Five example applications are presented in this paper: (1) Use of IRPhEP Data in Uncertainty Analyses and Cross Section Adjustment, (2) Uncertainty Evaluation Methods for Reactor Core Design at JAEA Using Reactor Physics Experimental Data, (3) Application of Benchmarking Data to a Broad Range of Criticality Safety Problems, (4) Cross Section Data Testing with ICSBEP Benchmarks, and (5) Use of the International Handbook of Evaluated Reactor Physics Benchmark Experiments to Support the Power Industry.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Verification of the AGREE and Serpent for the steady state HTR-10 benchmark problems

This paper provides results from steady state HTR-10 benchmark calculations performed using AGREE and Serpent that are compared to experimental results as well as calculations performed by INET. The purpose of completing this benchmark is to validate AGREE and Serpent for the prediction of HTGR operation so that they may ultimately be used to support licensing and deployment efforts of advanced reactors. The benchmark consists of several problems ranging from control rod worth calculations to k{sub eff} calculations at various temperatures. Overall, AGREE and Serpent show good agreement with the reference solutions and are effectively able to predict HTGR operation for a variety of steady state cases. The largest difference from the reference was for the initial core single rod worth, which is possibly due to the larger core helium cavity causing inaccuracies in the diffusion calculation whereas the control rod worth determined by Serpent and AGREE is much closer to the reference result for the full core loading. Moreover, in every benchmark problem, increasing the number of energy groups in the cross sections results in improved agreement of the AGREE result with the reference, although it also causes an increase in computation time. For problems with experimental data available, the accuracy of results generated by Serpent and AGREE is comparable to results obtained by other benchmark participants. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Validation of MCNP Critical Benchmark Models of Moderated Highly Enriched Uranium Slabs

A new centralized repository of high-quality MCNP models of critical benchmark experiments is currently under development at Los Alamos National Laboratory (LANL). The benchmark experiments are described in the International Criticality Safety Benchmark Evaluation Project (ICSBEP) Handbook, and the initial set of benchmark models are derived from the Whisper Suite provided with MCNP6.2. This effort is a collaboration among the Nuclear Criticality Safety, Nuclear Data, and Monte Carlo code development/application organizations at LANL. The goal is to build a single LANL benchmark collection that is up to date with the latest ICSBEP revision, has a formal review and revision process, is contained in an open-source repository, and utilizes new Python tools for improved input and output file review. This paper describes the validation of the models associated with HEU-MET-FAST-007, “Uranium Metal Slabs Moderated with Polyethylene, Plexiglas, and Teflon”. The Monte Carlo n-Particle (MCNP) models were compared to the second revision of HEU-MET-FAST-007. The experiment considered critical configurations of highly enriched uranium (HEU) metal slabs and various moderators in 43 unique cases. The slabs of uranium were separated by layers of moderating material, forming an assembly. The assembly was separated in two halves with one half placed on a stationary table and the other attached to a moveable table that could be raised and lowered via pulleys connected to the ceiling of the shielded room. To reach criticality, the two halves were brought together with a negligible gap between the assemblies.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Evaluation of Oak Ridge National Laboratory Health Physics Research Reactor Operation Data for Critical Benchmark Creation

The Oak Ridge National Laboratory (ORNL) Health Physics Research Reactor (HPRR) was a research reactor designed and built at ORNL in 1961. The critical assembly used a highly enriched uranium and molybdenum alloy as the fuel and could be operated in steady-state or burst modes. The HPRR has recently been the object of an investigation to create a criticality benchmark. Such benchmarks are very important, as they are used primarily to show the accuracy of newly developed modeling codes and to help experimental validation and reactor licensing. The evaluated experiments considered in this paper were carried out between 1974 and 1986 from various HPRR activities such as steady-state subcritical, steady-state critical, and burst prompt super-critical operations of the reactor for dosimetry, irradiation, or training purposes. By using the HPRR experimental logbook information and the as-built drawings of the critical assembly, a highly detailed model of the HPRR was created with SCALE 6.2.4/KENO-VI, and a first version of a critical benchmark of the HPRR was developed following the International Criticality Safety Benchmark Evaluation Project (ICSBEP) guidelines for thorough description and uncertainty/sensitivity quantification. Unfortunately, in most of the evaluated experiments, the obtained difference between calculated and experimental k eff is around 1,000 pcm, corresponding to a relative error of approximately 1%, beyond the quality standards of the ICSBEP recommending a relative error below 0.1%. Moreover, the derived experimental uncertainty is high, around 4% relative, mainly due to the U-Mo fuel density uncertainty, but also from numerous other factors. For these reasons, the creation of a valuable critical benchmark from HPRR operation data is thus far compromised. In this paper, the different steps of the experiments’ evaluation are summarized, and the reasons for the experimental/calculation discrepancies and potential ways to solve them are explored. This paper also aims to remind us always to exercise considerable care when performing experimental work, and to record all the data possible for potential future uses.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗