Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “BENCHMARKS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6
All sourcesNASA NTRSDOE OSTI
NASA NTRSDate not supplied

Benchmark Problem Development for Testing Maturity of Intelligent Contingency Management Tools

This paper presents a process used to develop appropriate scenarios and metrics for evaluating the maturity of intelligent contingency management algorithms. A benchmark scenario is a reference point against which something can be measured, compared, or assessed. Creating an accurate benchmark requires considerable research and expertise. The scenario itself is an artificial representation of a real-world event, designed to achieve a set of learning objectives through experiential learning. Designing an effective benchmark simulation scenario requires careful planning, including identification of clear objectives; capability assessment of the algorithm/tool being evaluated; assessment of necessary levels of fidelity; development of a process flow map of events and event interactions; and identification of metrics that map back to objectives. Thus, a benchmark scenarios for contingency management might consist of one or several commonly used functions taken from real world applications, used for evaluation, characterization and performance measurement of a contingency management algorithm. Behavior of the contingency management algorithm under different environmental conditions should then be able to be predicted using a set of benchmark functions. The paper describes the resulting benchmark problem as an illustration of the application of this process.

Jon Holbrook↗
DOE OSTI2022

Analytical benchmark solution for 1-D neutron transport coupled with thermal conduction and material expansion

In this work we present fully analytical solutions for a class of finite, homogeneous, 1-D slab benchmark problems with nonlinear temperature feedback effects. The proposed class of benchmarks include multiplicative 1-D neutron transport (limited to quasistatic S{sub 2} with μ = ±1) coupled with thermal conduction, convection, Doppler broadening, and expansion effects along the length of the slab. This class of benchmark models, along with the corresponding analytical solutions, are valuable for validating multiphysics analysis frameworks that support coupled neutronics/thermal/structural calculations. Analytical solutions for the benchmarks are obtained by introducing an ansatz that the equilibrium flux and temperature distributions in the slab have the same shape. Specific values for the total microscopic cross section, σ{sub t,0}, and conductive heat transfer coefficient, h, that satisfy the assumed ansatz are then determined. A discussion of the procedure for generating benchmark models and analytical solutions is provided, along with numerical results for an example set of model parameters and thoughts on practical applications of the benchmarks for multiphysics code validation. (authors)

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

Status of the International Criticality Safety Benchmark Evaluation Project

The International Criticality Safety Benchmark Evaluation Project (ICSBEP) has continued its work generating evaluations of new and historical benchmark experiments since the last update to the nuclear criticality safety (NCS) community at the 12th International Conference on Nuclear Criticality Conference held in 2023. One additional version of the ICSBEP Handbook has been published since that update, and the Technical Review Group (TRG) held two in-person meetings to review and approve additional benchmarks. The 2022 and 2023 editions of the handbook were combined into one release (published in November 2024) and contained 13 new evaluations with 46 different configurations and two major revisions to existing evaluations. The 2024 version of the handbook, currently under publication review, will contain two new evaluations with 15 new configurations and one major revision to HEU-MET-FAST-028, the evaluation of Flattop with a uranium core. The ICSBEP TRG met again in person in April 2025 to review benchmarks for the 2025 ICSBEP Handbook and final comment resolution is currently ongoing. Many of the new benchmarks represent contemporaneous experiments that have been specifically optimized to provide validation cases relevant to the NCS community. One major area of focus for new critical experiments is to target the sparsely populated intermediate energy (or resonance) region. Another focus of many of the new benchmarks is to provide experiments sensitive to different materials, such as chlorine, hafnium, tantalum, titanium, molybdenum, chromium, and polymethyl methacrylate (PMMA, or Lucite). The ICSBEP continues to deliver high-quality, peer reviewed evaluations of integral experiments relevant to the nuclear data community.

HEU-MET-FAST-028↗
DOE OSTI2026

Intern-Artificial Intelligence Benchmarking

Benchmarks provide a standardized method for evaluating different AI models, enabling reproducibility and comparison between models, and facilitating scientific progress. As AI models continue to develop rapidly, incorporating new datasets, capabilities, and architectures becomes more complicated. Therefore, the current static benchmarks become increasingly irrelevant. The MLCommons team argues that to make AI benchmarks more relevant, it involves making the benchmarks themselves more dynamic, as well as technical innovations that make it easier for scientists and researchers at all levels to use and contribute to the benchmarks. The current progress in technical innovation is a software that allows for a detailed view of a collection of AI benchmarks to be output in various formats that are easily readable and accessible.

Krishnan, Anjay [Fermilab]↗
DOE OSTI2025

Classifying and rating AI benchmarks

We created a set of standards to efficiently evaluate AI benchmarks through objective means. Although prevalent, especially in recent times, AI benchmarks have no single way to measure their effectiveness. The MLCommons team provided a set of criteria for evaluating benchmarks, although the criteria lacks a clearly defined set of evaluation rules. We created a rubric with preset factors to efficiently and objectively evaluate a benchmark’s quality. We created a software framework for processing lists of benchmarks for visualization. The framework and rating system allows researchers to quickly check if their benchmarks are effective.

Shiraishi, Reece [Fermilab]↗
DOE OSTI2023

Status of the International Criticality Safety Benchmark Evaluation Project

The International Criticality Safety Benchmark Evaluation Project (ICSBEP) has continued its work generating evaluations of new and historical criticality benchmark experiments since the last update to the nuclear criticality safety community at the 11th International Conference on Nuclear Criticality Safety (ICNC 2019) in Paris, France. Three additional versions of the ICSBEP Handbook have been published since that update, and the Technical Review Group (TRG) held two in-person (in 2019 and 2023) and three virtual (2020 and 2021) meetings to review and approve additional benchmarks. The 2019 edition of the ICSBEP Handbook included five new evaluations with 79 new configurations, the 2020 version of the ICSBEP Handbook contained five new evaluations totaling 76 new configurations, and the 2021 version of the ICSBEP handbook contained five new evaluations with a total of 57 different configurations. The ICSBEP TRG met in October and December 2021, to review benchmarks for potential inclusion in the 2022 ICSBEP Handbook, with seven evaluations receiving provisional approval pending resolution of review group comments. Final comment resolution for some of these evaluations is currently underway and handbook publication should be completed soon. The ICSBEP TRG met again in person in April 2023 to review benchmarks for the 2023 ICSBEP Handbook, provisionally approving 7 new evaluations. The ICSBEP continues to deliver high-quality, peer reviewed evaluations of experiments relevant to the nuclear criticality safety community.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

Autotuning PolyBench benchmarks with LLVM Clang/Polly loop optimization pragmas using Bayesian optimization

Here, we develop a ytopt autotuning framework that leverages Bayesian optimization to explore the parameter space search and compare four different supervised learning methods within Bayesian optimization and evaluate their effectiveness. We select six of the most complex PolyBench benchmarks and apply the newly developed LLVM Clang/Polly loop optimization pragmas to the benchmarks to optimize them. We then use the autotuning framework to optimize the pragma parameters to improve their performance. The experimental results show that our autotuning approach outperforms the other compiling methods to provide the smallest execution time for the benchmarks syr2k, 3mm, heat-3d, lu, and covariance with two large datasets in 200 code evaluations for effectively searching the parameter spaces with up to 170,368 different configurations. We find that the Floyd-Warshall benchmark did not benefit from autotuning. To cope with this issue, we provide some compiler option solutions to improve the performance. Then we present loop autotuning without a user's knowledge using a simple mctree autotuning framework to further improve the performance of the Floyd-Warshall benchmark. We also extend the ytopt autotuning framework to tune a deep learning application.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Defining and applying an electricity demand flexibility benchmarking metrics framework for grid-interactive efficient commercial buildings

Building demand flexibility (DF) research has recently gained attention. To unlock building DF as a predictable grid resource, we must establish a quantitative understanding of the resource size, performance variability, and predictability based on large empirical datasets. Researchers have proposed various sets of theoretical metrics to measure this performance. Some metrics have been applied to simulation results, but most fall short of exploring the complexities in real building applications. There are practical metrics used in individual demand response field studies but they alone cannot fulfil the job of DF benchmarking across a diverse group of buildings. The electrical grid's geographically diverse and changing nature presents challenges to comparing building DF performance measured under different conditions (i.e., benchmarking DF). To address this challenge, a novel DF benchmarking framework focused on load shedding and shifting is presented; the foundation is a set of simple, proven single-event metrics with attributes describing event conditions. These enable benchmarking and visualization in different dimensions for identifying trends that represent how these attributes influence DF. To test its feasibility and scalability, the DF framework was applied to two case studies of 11 office buildings and 121 big-box retail buildings with demand response participation data. Furthermore, these examples provided a pathway for using both building level benchmarking and aggregation to extract insights into building DF about magnitude, consistency, and influential factors. Potential applications of the framework and real-world values have been identified for grid and building stakeholders.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Benchmark of the Kilowatt Reactor Using Stirling TechnologY (KRUSTY) Component Critical Configurations

Kilowatt Reactor Using Stirling TechnologY (KRUSTY) was a prototype for the U.S. National Aeronautics and Space Administration’s Kilopower Program. KRUSTY has a highly enriched uranium–molybdenum alloy (with 7.65 wt% molybdenum) annular core reflected by beryllium oxide with an outer stainless steel shield. Five configurations from the experimental campaign were chosen to be evaluated as benchmark cases. Uncertainties were evaluated in five categories: (1) criticality measurement, (2) mass and density, (3) dimensions, (4) material compositions, and (5) positioning. The largest contribution to the overall uncertainty in each case was from the radial alignment of the movable platen. A simplified model was created to increase computational efficiency, and an average bias of –16 pcm was calculated due to the simplifications. Sample calculations were completed for each case using MCNP6.2, COG, and MC21, all with ENDF/B-VIII.0 nuclear data. For MCNP6.2, the average difference (absolute value) between the calculated and experimental $k_{eff}$ for the five configurations was 14 pcm for both the detailed and the simplified models. The $k_{eff}$ results from all three codes are within 1σ of the benchmark values. KRUSTY’s value as a benchmark is due to its sensitivity to beryllium and molybdenum. For beryllium, KRUSTY adds an 18th benchmark with a total cross-section sensitivity greater than 0.05%/%/(unit lethargy). For molybdenum, KRUSTY adds a 9th benchmark with a total cross-section sensitivity greater than 0.004%/%/(unit lethargy).

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2024

The MUSIC Critical Benchmark and Nuclear Data

The Measurement of Uranium Subcritical and Critical (MUSIC) experiment was a series of measurements of critical and subcritical configurations of bare highly enriched uranium. The goal was to compare measurement methods, analysis techniques, and simulation methods across regimes of criticality and to provide high-quality validation of 235 U nuclear data. A benchmark evaluation of the two critical configurations of the MUSIC experiments will soon be published in the release of the International Criticality Safety Benchmark Evaluation Project Handbook. The recent execution of the experiment aids in proper quantification of model simplifications and all uncertainties associated with the experiment. Historical benchmark evaluations are heavily relied on for uranium nuclear data validation despite the fact that the same level of documentation and comparable uncertainty analysis may not be present. The MUSIC evaluation is less likely to include “unknown unknowns” that could impede accurately modeling the system. Presented are both highly detailed and very simplified models, which represent the experimental configurations accurately, aiding the users of the benchmark for nuclear data or transport code validation. The sensitivities of k eff to nuclear data and nuclear data–related uncertainties are very similar between this experiment and previous bare uranium sphere experiments. In addition, the nuclear data uncertainties to any nuclides other than 235 U are small. For all these reasons, the recently evaluated MUSIC benchmark critical configurations could prove very useful for 235 U nuclear data validation. Currently, major libraries have good agreement with the experimental results, within 200 pcm for all nuclear data libraries, and within one standard deviation of the experimental result for most. Suggested nuclear data adjustments based on MUSIC and Lady Godiva are also presented, with posterior improvements to both the agreement in k eff and the uncertainty associated with the nuclear data.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

An Analytic Benchmark for Neutron Boltzmann Transport with Downscattering—Part IV: PFNS and $\bar{ν}$ Uncertainty Propagation

An analytic benchmark with continuous-energy cross sections was previously derived to validate criticality calculations. Here, to extend the utility of the analytic benchmark to verify the implementation of $\bar{ν}$ and prompt fission neutron spectrum (PFNS) uncertainty propagation methods, new simplified forms that are dependent on the incident (fission-causing) neutron energy, as well as the outgoing neutron energy for the PFNS, are introduced in this work. The analytical forms for the flux and adjoint flux are derived for the extended benchmark and used to determine the 𝑘-eigenvalue sensitivity to $\bar{ν}$ and PFNS. The 𝑘-eigenvalue uncertainty due to $\bar{ν}$ and PFNS is calculated for the analytic benchmark using simplified$\bar{ν}$ and PFNS representations based on the ENDF-B/VIII.0 239 Pu evaluation. Because of the low sensitivity of the analytic benchmark to the physical PFNS, a nonphysical high-sensitivity PFNS is also presented.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

Random circuit block-encoded matrix and a proposal of quantum LINPACK benchmark

The LINPACK benchmark reports the performance of a computer for solving a system of linear equations with dense random matrices. Although this task was not designed with a real application directly in mind, the LINPACK benchmark has been used to define the list of TOP500 supercomputers since the debut of the list in 1993. We propose that a similar benchmark, called the quantum LINPACK benchmark, could be used to measure the whole machine performance of quantum computers. The success of the quantum LINPACK benchmark should be viewed as the minimal requirement for a quantum computer to perform a useful task of solving linear algebra problems, such as linear systems of equations. We propose an input model called the Random Circuit Block-Encoded Matrix (RACBEM), which is a proper generalization of a dense random matrix in the quantum setting. The RACBEM model is efficient to be implemented on a quantum computer and can be designed to optimally adapt to any given quantum architecture, with relying on a black-box quantum compiler. Besides solving linear systems, the RACBEM model can be used to perform a variety of linear algebra tasks relevant to many physical applications, such as computing spectral measures, time series generated by a Hamiltonian simulation, and thermal averages of the energy. We implement these linear algebra operations on IBM Q quantum devices as well as quantum virtual machines, and demonstrate their performance in solving scientific computing problems.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

Verification of the AGREE and Serpent for the steady state HTR-10 benchmark problems

This paper provides results from steady state HTR-10 benchmark calculations performed using AGREE and Serpent that are compared to experimental results as well as calculations performed by INET. The purpose of completing this benchmark is to validate AGREE and Serpent for the prediction of HTGR operation so that they may ultimately be used to support licensing and deployment efforts of advanced reactors. The benchmark consists of several problems ranging from control rod worth calculations to k{sub eff} calculations at various temperatures. Overall, AGREE and Serpent show good agreement with the reference solutions and are effectively able to predict HTGR operation for a variety of steady state cases. The largest difference from the reference was for the initial core single rod worth, which is possibly due to the larger core helium cavity causing inaccuracies in the diffusion calculation whereas the control rod worth determined by Serpent and AGREE is much closer to the reference result for the full core loading. Moreover, in every benchmark problem, increasing the number of energy groups in the cross sections results in improved agreement of the AGREE result with the reference, although it also causes an increase in computation time. For problems with experimental data available, the accuracy of results generated by Serpent and AGREE is comparable to results obtained by other benchmark participants. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2020

Validation of MCNP Critical Benchmark Models of Moderated Highly Enriched Uranium Slabs

A new centralized repository of high-quality MCNP models of critical benchmark experiments is currently under development at Los Alamos National Laboratory (LANL). The benchmark experiments are described in the International Criticality Safety Benchmark Evaluation Project (ICSBEP) Handbook, and the initial set of benchmark models are derived from the Whisper Suite provided with MCNP6.2. This effort is a collaboration among the Nuclear Criticality Safety, Nuclear Data, and Monte Carlo code development/application organizations at LANL. The goal is to build a single LANL benchmark collection that is up to date with the latest ICSBEP revision, has a formal review and revision process, is contained in an open-source repository, and utilizes new Python tools for improved input and output file review. This paper describes the validation of the models associated with HEU-MET-FAST-007, “Uranium Metal Slabs Moderated with Polyethylene, Plexiglas, and Teflon”. The Monte Carlo n-Particle (MCNP) models were compared to the second revision of HEU-MET-FAST-007. The experiment considered critical configurations of highly enriched uranium (HEU) metal slabs and various moderators in 43 unique cases. The slabs of uranium were separated by layers of moderating material, forming an assembly. The assembly was separated in two halves with one half placed on a stationary table and the other attached to a moveable table that could be raised and lowered via pulleys connected to the ceiling of the shielded room. To reach criticality, the two halves were brought together with a negligible gap between the assemblies.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2022

Evaluation of Oak Ridge National Laboratory Health Physics Research Reactor Operation Data for Critical Benchmark Creation

The Oak Ridge National Laboratory (ORNL) Health Physics Research Reactor (HPRR) was a research reactor designed and built at ORNL in 1961. The critical assembly used a highly enriched uranium and molybdenum alloy as the fuel and could be operated in steady-state or burst modes. The HPRR has recently been the object of an investigation to create a criticality benchmark. Such benchmarks are very important, as they are used primarily to show the accuracy of newly developed modeling codes and to help experimental validation and reactor licensing. The evaluated experiments considered in this paper were carried out between 1974 and 1986 from various HPRR activities such as steady-state subcritical, steady-state critical, and burst prompt super-critical operations of the reactor for dosimetry, irradiation, or training purposes. By using the HPRR experimental logbook information and the as-built drawings of the critical assembly, a highly detailed model of the HPRR was created with SCALE 6.2.4/KENO-VI, and a first version of a critical benchmark of the HPRR was developed following the International Criticality Safety Benchmark Evaluation Project (ICSBEP) guidelines for thorough description and uncertainty/sensitivity quantification. Unfortunately, in most of the evaluated experiments, the obtained difference between calculated and experimental k eff is around 1,000 pcm, corresponding to a relative error of approximately 1%, beyond the quality standards of the ICSBEP recommending a relative error below 0.1%. Moreover, the derived experimental uncertainty is high, around 4% relative, mainly due to the U-Mo fuel density uncertainty, but also from numerous other factors. For these reasons, the creation of a valuable critical benchmark from HPRR operation data is thus far compromised. In this paper, the different steps of the experiments’ evaluation are summarized, and the reasons for the experimental/calculation discrepancies and potential ways to solve them are explored. This paper also aims to remind us always to exercise considerable care when performing experimental work, and to record all the data possible for potential future uses.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
DOE OSTI2023

Engagement opportunities in OECD NEA benchmark development

A myriad of opportunities is available to collaborate via international benchmark exercises and experimental data preservation activities. Many such opportunities abound under the auspices of the Nuclear Science Committee of the Organisation for Economic Co-operation and Development Nuclear Energy Agency (NEA). Key projects and activities of relevance to the development of advanced reactors design include the International Criticality Safety Benchmark Evaluation Project (ICSBEP), the International Reactor Physics Experiment Evaluation Project (IRPhEP), the International Assay Data of Spent Nuclear Fuel Database (SFCOMPO), the Shielding Integral Benchmark and Archive Database (SINBAD), and The International Experimental Thermal HYdraulicS Database (TIETHYS), and various cooperative benchmark exercises. Interested participants are encouraged to contact the leadership and secretariat of the various Technical Working Groups and Working Parties to become more engaged. This paper provides a summary of the current benchmark exercises and experimental databases available for international participation.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2023

Intrinsic value of the international benchmark projects, ICSBEP and IRPhEP, for advanced reactor development

The International Criticality Safety Benchmark Evaluation Project, a sanctioned program under the auspices of the Nuclear Energy Agency of the Organisation for Economic Co-operation and Development, has been a highly successful and productive collaboration, now encompassing over 5,000 evaluated experimental benchmarks trusted and relied upon throughout the international nuclear communities. The success of this project led to the development of the International Reactor Physics Experiment Evaluation Project, which is dedicated to the evaluation of benchmark experiment data to sustain current and future reactor physics validation needs. These exemplary programs, and their widely utilized handbooks, serve as gold standards to which other databases strive to emulate. The purpose of the two projects is to preserve modern and legacy experimental data and evaluate it in a standardized handbook format to provide quality benchmarks to support modern and future criticality safety and reactor physics validation. These two projects have often served as the mechanism through which historic and modern neutronics experiments are evaluated and shared across international borders, to best provide unique, high-quality peer-reviewed, and often otherwise unavailable, benchmark data. The contents of these handbooks are utilized not only in validating criticality safety, reactor physics, and advanced reactor calculations, but are used to validate neutronics calculations and nuclear data for most other nuclear applications. This paper discusses both international programs and available content to enable advanced reactor design validation.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
NASA NTRSDate not supplied

Spacecraft Observatory Benchmark Problem for Optical Disturbance Rejection

Since mid-1980s NASA has launched several first-generation interplanetary observatory missions. A salient feature of these missions has been the mounting of the optical payload on top of a large flexible space structure. This imposes challenging control-structure interaction problems (CSI) that have been studied over several decades. The Dec. 2021 launch of the James Webb Telescope marks the latest example of such a mission. One way to encourage development of novel control methods is to provide a benchmark problem for researchers. By the mid-2000s, there were a few well-known benchmark problems developed by the control community with a focus on CSI applications. However, in the past 15 years, the control challenges in optical observation missions have shifted from CSI to line-of-sight (LOS) precision pointing. This is because popular designs of the second-generation (Gen II) space earth missions have eliminated the large structure interface by mounting the optical payload directly on top of a rigid bus platform. The focus now is to achieve accurate pointing with maximum rejection of the surrounding disturbances. The NASA Safety and Engineering Center (NESC) has tasked T he Aerospace Corporation to develop a Benchmark Problem to study the challenging control issues that Gen II observatory missions encounter. This software tool is to be released as a public domain platform aimed at government, academic, and industry researchers. The focus of this benchmark problem is for researchers to design a set of innovative payload control laws with the maximum Optical Disturbance Rejection (ODR) capability to meet a set of pre-defined μrad-level of LOS jitter requirements. This paper describes the development of the benchmark problem. Aerospace/NASA will provide an integrated LOS plant model and disturbance/command profiles for users to run their design and simulation with as well as a user’s guide describing the required interfaces. The Aerospace Corp is also working to develop a hardware fast steering mirror testbed for users to demonstrate their innovative design in real hardware, if so desired.

Spacecraft Observatory Benchmark Problem↗