Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Updated Godiva-IV Benchmark Preview

A note of errata prepended to the Godiva-IV delayed-critical benchmark (HEU-MET-FAST- 086) identifies two corrections to be made to the model: The glory hole in the spindle should be made larger, and the height of the safety block should be made smaller (and therefore its density made larger). In addition, the safety block at its full-in position is closer to the inner subassembly plate than was modeled in the benchmark. These changes have been made to HEU-MET-FAST-086 Case 4 in order to estimate the effect on $\kappa$ eff and on the neutron flux spectrum. Using smaller separation, a smaller safety block, and a larger glory hole caused keff to increase by 453 ± 1 pcm from the benchmark. The latest nuclear data, ENDF/B-VIII.0, have also been used, causing $\kappa$ eff to increase another 37 ± 1 pcm. Flux spectra were compared in a modeled fission foil (near the center of Godiva-IV in the glory hole) and at three external (point) detectors. Within the fission foil, using ENDF/B-VIII.0 induced changes in the flux spectrum similar in size to the changes due to using smaller separation, a smaller safety block, and a larger glory hole. At the detectors, using ENDF/B- VIII.0 induced changes in the flux spectrum much larger than those due to changing the model. In other words, the corrections to the benchmark model cause a large increase in $\kappa$ eff , but the changes to the neutron flux spectrum are small compared to those caused by using the latest nuclear data. This study presents a preview of results expected during the reevaluation of the Godiva IV benchmark, but it is not a substitute for the full reevaluation.A note of errata prepended to the Godiva-IV delayed-critical benchmark (HEU-MET-FAST- 086) identifies two corrections to be made to the model: The glory hole in the spindle should be made larger, and the height of the safety block should be made smaller (and therefore its density made larger). In addition, the safety block at its full-in position is closer to the inner subassembly plate than was modeled in the benchmark. These changes have been made to HEU-MET-FAST-086 Case 4 in order to estimate the effect on $\kappa$ eff and on the neutron flux spectrum. Using smaller separation, a smaller safety block, and a larger glory hole caused $\kappa$ eff to increase by 453 ± 1 pcm from the benchmark. The latest nuclear data, ENDF/B-VIII.0, have also been used, causing $\kappa$ eff to increase another 37 ± 1 pcm. Flux spectra were compared in a modeled fission foil (near the center of Godiva-IV in the glory hole) and at three external (point) detectors. Within the fission foil, using ENDF/B-VIII.0 induced changes in the flux spectrum similar in size to the changes due to using smaller separation, a smaller safety block, and a larger glory hole. At the detectors, using ENDF/B- VIII.0 induced changes in the flux spectrum much larger than those due to changing the model. In other words, the corrections to the benchmark model cause a large increase in $\kappa$ eff , but the changes to the neutron flux spectrum are small compared to those caused by using the latest nuclear data. This study presents a preview of results expected during the reevaluation of the Godiva IV benchmark, but it is not a substitute for the full reevaluation.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2020

Validation and Independent Uncertainty Analysis of the MIX-SOL-THERM-003 ICSBEP Benchmark

The International Criticality Safety Benchmark Evaluation Project (ICSBEP) was started in 1992 by the United States Department of Energy and later in 1995 became an international project with contributions from 22 countries. The project is now organized by the OECD (Organisation for Economic Co-operation and Development) Nuclear Energy Agency (NEA). In its most recent iteration, the ICSBEP handbook contains over five thousand evaluations of critical, near-critical, and subcritical experiments conducted in facilities all around the world. These benchmarks serve as valuable information for criticality safety engineers who can use them to validate calculation techniques and establish minimum subcritical margins for operations with fissionable materials. The benchmarks in the handbook are categorized by their fissile material composition, material form (oxide, solution, or metal), and fission energy spectra. This is especially useful for those looking for benchmarks similar to a system they are working on to compare methods and identify trends. The ICSBEP Handbook Uncertainty Guide is document outlining recommended practices and methods for determining uncertainties in these benchmarks. Quantifying these uncertainties thoroughly is crucial as it allows a higher degree of confidence that data used from them is valid and relevant. The guide stresses the importance of a thorough and well documented uncertainty analysis when evaluating an experiment. All measured values of a system, whether they be dimensions or material compositions, have a certain amount of uncertainty associated with them and can be analyzed one by one to determine their effects on the system. Many evaluated benchmarks in the handbook present this in detail, however some do not, mostly earlier evaluations performed in the 1990’s and early 2000’s. Recently at Los Alamos National Laboratory (LANL), the Nuclear Criticality Safety Division (NCSD) of LANL has been validating MCNP6.2 ® input files of criticality benchmarks for use by Whisper, a criticality safety code developed at LANL. This effort is also part of the OECD NEA Working Party on International Nuclear Data Evaluation Co-operation (WPEC) Subgroup 45, also known as Validation of Nuclear Data Libraries (VaNDaL). The goal of VaNDaL is to compile a set of validated simulation inputs for use in validating nuclear data and simulation codes. As part of these efforts, one of the benchmarks reviewed was the MIX SOL-THERM-003 ICSBEP benchmark. This paper provides an independent uncertainty analysis of this benchmark experiment.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

U.S. Solar Photovoltaic System and Energy Storage Cost Benchmark: Q1 2021 [Slides]

Our benchmarking method includes bottom-up accounting for all necessary system and projectdevelopment costs incurred when installing residential, commercial, and utility-scale systems, and it models the Q1 2021 costs for such systems, excluding any previous supply agreements or contracts. In general, we attempt to model the typical installation techniques and business operations from an installed-cost perspective, and our benchmarks are national averages. The residential PV-only benchmark and the commercial rooftop PV-only benchmark average costs by inverter type (string inverters, string inverters with direct current [DC] optimizers, and microinverters), weighted by inverter market share. The residential PV-only benchmark is further averaged across small installer and national integrator business models, weighted by market share. All benchmarks include variations—accounting for the differences in size, equipment, and operational use (particularly for storage)—that are currently available in the marketplace. All benchmarks assume nonunionized construction labor; residential and commercial PV systems predominantly use nonunionized labor, and the type of labor required for utility-scale PV systems depends heavily on the development process. All benchmarks assume the use of monofacial monocrystalline silicon PV modules. Benchmarks using cadmium telluride or bifacial modules could result in significantly different results. The data in this annual benchmark report inform the formulation of and track progress toward the U.S. Department of Energy (DOE) Solar Energy Technologies Office’s Government Performance and Reporting Act cost targets.

14 SOLAR ENERGY↗
DOE OSTI2022

Summary of LANL Critical Benchmark Comparison Study and Revisions for Cases Involving HEU, LEU, MIX, and Pu

This report documents results obtained for revisions made to cases involving Highly Enriched Uranium (HEU), Intermediate Enriched Uranium (IEU), a mixture of Pu and Uranium (MIX), as well as Pu cases. A previous summary of revisions for HEU an Pu cases was reported and additional investigations into four cases originally presented therein uncovered further revisions which led to better agreement with other transport codes, those cases are updated in this report. The summary of all cases reported in Reference 2 is updated in this report. In addition, a previous summary of revisions for LEU and MIX was reported, a summary of those revisions in reproduced in this report for a comprehensive summary of changes to benchmarks beginning in fiscal year 2020 to current date. The report focuses on the changes made to LANL benchmarks modeled with MCNP6 using ENDF/B-VII.1 nuclear data that appeared to have discrepant results when compared with results of other codes. Feedback was used to pinpoint review of benchmark input files and to revise them when necessary. This report documents the results of review and revision of specific benchmarks highlighted as possibly discrepant in the comparison study. In addition, there is an effort tied to this work involving collaboration between LANL XCP and NCS Divisions in the development of a shared review/revision procedure and use of a new benchmark repository. LANL has a benchmark library of critical experiments from the International Criticality Safety Benchmark Evaluation Project (ICSBEP) Handbook modeled for use with MCNP. This collection is now over 1100 benchmarks, referred to as the Whisper-1.1 library because it is used with the sensitivity/uncertainty package, Whisper, which supports nuclear criticality safety validation and is released with MCNP6.2. The collection, originally created several decades ago, is a combination of smaller collections, which has been revised and expanded, by various groups at LANL over the years. The original authors are no longer at the laboratory and little formal documentation of review and revision of these benchmarks exists today. A branch of the benchmark collection was already the subject of a formal review undertaken by the LANL NCS Division and expanded to include XCP Division.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2025

An MLCommons Scientific Benchmarks Ontology

Scientific machine learning research spans diverse domains and data modalities, yet existing benchmark efforts remain siloed and lack standardization. This makes novel and transformative applications of machine learning to critical scientific use-cases more fragmented and less clear in pathways to impact. This paper introduces an ontology for scientific benchmarking developed through a unified, community-driven effort that extends the MLCommons ecosystem to cover physics, chemistry, materials science, biology, climate science, and more. Building on prior initiatives such as XAI-BENCH, FastML Science Benchmarks, PDEBench, and the SciMLBench framework, our effort consolidates a large set of disparate benchmarks and frameworks into a single taxonomy of scientific, application, and system-level benchmarks. New benchmarks can be added through an open submission workflow coordinated by the MLCommons Science Working Group and evaluated against a six-category rating rubric that promotes and identifies high-quality benchmarks, enabling stakeholders to select benchmarks that meet their specific needs. The architecture is extensible, supporting future scientific and AI/ML motifs, and we discuss methods for identifying emerging computing patterns for unique scientific workloads. The MLCommons Science Benchmarks Ontology provides a standardized, scalable foundation for reproducible, cross-domain benchmarking in scientific machine learning. A companion webpage for this work has also been developed as the effort evolves: https://mlcommons-science.github.io/benchmark/

Hawks, Ben [Fermilab] (ORCID:0000000157000288)↗
DOE OSTI2020

Gaussian Process Optimization of Sensitivity-Based Similarity Metrics between New Nuclear Applications and New/Existing Benchmarks

Confidently designing safe, new nuclear criticality experiments requires expert judgement, which could take years of experience. Sensitivity/uncertainty (S/U) analysis can be utilized by less experienced individuals to conservatively estimate uncertainties in important parameters, such as k eff , in newly proposed nuclear applications. This type of analysis relies on matching new nuclear applications with existing benchmark experiments. The Whisper-1.1 software package included in MCNP6.2 ®1 contains more than 1,100 International Criticality Safety Benchmark Evaluation Project (ICSBEP) benchmarks. These benchmarks however rarely match new nuclear applications. The number of benchmarks available to match a given set of materials or geometric configurations varies significantly. Furthermore, recently performed benchmark experiments may not have had enough time to be properly documented and published. Benchmarks are vital for determining the accuracy of nuclear data and can assist nuclear physics and evaluators in improving nuclear data libraries. Exhaustively exploring the parameter space using simulations with software such as MCNP is too computationally expensive. In this work, Gaussian process optimization was implemented to reduce the number of simulations needed for optimization over multiple parameters. This optimization scheme was designed to selectively generate new benchmarks with high sensitivity-based similarity metrics to user-defined nuclear applications. Two benchmark models of spherically nested shells containing plutonium, uranium, tantalum, and water were used in the optimization to match an application containing plutonium plates, stacked in a tantalum reflector, surrounded by water.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2022

Sensitivity and uncertainty of the IFR-1 BISON benchmark

The fuel performance code BISON is being used to evaluate metallic fuel for a new fast-spectrum test reactor called the Versatile Test Reactor, which is being considered for adoption by the US Department of Energy. To quantify the accuracy of BISON predictions, researchers at Oak Ridge National Laboratory have been developing a series of benchmarks based on legacy metallic fuel experiments. As part of this effort, the sensitivity of BISON predictions to variations in model inputs and the uncertainties associated with BISON predictions must be established. This paper summarizes efforts to perform a comprehensive sensitivity analysis (SA) and uncertainty quantification (UQ) on a benchmark based on the IFR-1 experiment.For the SA, at least one input was chosen from every BISON model and physics module used in the benchmark. The inputs were varied individually in a series of BISON simulations. Here, the resulting variations in benchmark predictions were normalized to calculate sensitivities. These sensitivities were then used to inform input selections for the UQ.The UQ was performed using the Monte Carlo UQ method. A literature review was conducted to estimate uncertainty distributions for the selected inputs, and values were sampled randomly from each distribution in a series of BISON simulations. Variations in the benchmark predictions were used to estimate uncertainty distributions and confidence intervals. It was found that nearly 100% of the benchmark predictions matched the corresponding legacy values within the confidence intervals. However, this is at least partially because the confidence intervals associated with benchmark predictions were wide. The uncertainty contributions of assumptions in the benchmark, experimental uncertainties, and BISON models were quantified. Some analysis was performed to identify inputs that contributed to the uncertainties. Finally, recommendations are made for future benchmark and future BISON development.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2026

Informing Robust Functional Relationship Benchmarks: An Evaluation of the Temperature Sensitivity of Ecosystem Respiration Across the Arctic-Boreal Region

During land model development, simulated carbon dynamics are often benchmarked against observational data sets to evaluate model performance. Functional relationship benchmarks are the relationship between a driving variable (e.g., temperature) and a response variable (e.g., ecosystem respiration) and are a promising tool for assessing model performance by evaluating modeled sensitivities to changing environmental conditions. However, observed functional relationships can be influenced by choices made during data collection and throughout the benchmarking process, impacting the inferred skill of land models. To avoid misrepresenting a model's true performance, it is necessary to systematically evaluate best practices when constructing functional relationship benchmarks. We developed a set of guidelines for constructing functional relationship benchmarks, considering the choice of data set, number of daily observations, temporal extent, and temporal resolution across Alaska and Canada over a 20-year period from 2001 to 2020. The temperature sensitivity of ecosystem respiration from observations, evaluated through an apparent Q 10 , is highly variable both spatially and as a result of the data processing approach applied in the benchmark formation. When benchmarking 13 models from the Warming Permafrost Model Intercomparison Project (WrPMIP), the range in inferred model skill is substantially impacted by the choices applied in constructing functional relationship benchmarks. The inferred performance of a given model is most sensitive to the number of daily observations and temporal extent, followed by choice of benchmark data set and temporal averaging. Results from this analysis can guide the development of consistent and robust functional relationships for future model evaluation studies.

Poe, Jeralyn [Northern Arizona University, Flagsta↗
DOE OSTI2025

Benchmarking quantum computers

The rapid pace of development in quantum computing technology has sparked a proliferation of benchmarks to assess the performance of quantum computing hardware and software. However, not all benchmarks are of equal merit. Good ones empower scientists, engineers, programmers and users to understand the power of a computing system, whereas bad ones can misdirect research and inhibit progress. In this Perspective, we survey the science of quantum computer benchmarking. Here, we discuss the role of benchmarks and benchmarking and how good benchmarks can drive and measure progress towards the long-term goal of useful quantum computations, known as quantum utility. We explain how different kinds of benchmark quantify the performance of different parts of a quantum computer, discuss existing benchmarks, examine recent trends in benchmarking, and highlight important open research questions in this field.

Proctor, Timothy James [Sandia National Laboratori↗
DOE OSTI2023

Benchmarking for AI for Science

AI has been instrumental for recent developments in a number of domains of the sciences. With several hundred machine learning (ML) algorithms and models, and numerous AI-specific hardware platforms, a common quest for all scientists working on AI for Science is around the selection of machine learning algorithm(s) to solve their domain-specific scientific problems. A number of different initiatives around AI Benchmarking have been set up and have been useful in understanding the benefits of different ML algorithms for different tasks.However, with the majority of these AI Benchmarking initiatives focusing on the conventional notions of benchmarking, where the focus is purely runtime performance (such as training time or inference time), their suitability for benchmarking different ML algorithms for solving scientific problems has been viewed as a performance problem even though both are hardly the same. To make reasonable, explainable, and justifiable advancements in science using AI, it is critical to focus on the merits of these algorithms in handling different domain science problems. In other words, more emphasis must be given on Benchmarking for AI for Science than AI Benchmarking. The vision of the former is not only to assess the performance of ML algorithms, but also to assess, and understand the benefits and merits of different ML algorithms in handling scientific problems. Benchmarking for AI for Science, instead of pure performance focused AI Benchmarking, has several benefits: (i) it has the potential to offer advances in the sciences, much more rapidly than through pure performance-based AI methods, (ii) it will encourage the community to focus on developing better domain-specific AI techniques, particularly given the provision for being able to benchmark different techniques, and (iii) it will encourage hardware manufacturers to focus on developing science-specific hardware subsystems.

Thiyagalingam, Jeyan↗
DOE OSTI2018

Entity Resolution at Large Scale: Benchmarking and Algorithmics

We seek scalable benchmarks for entity resolution problems. Solutions to these problems range from trivial approaches such as string sorting to sophisticated methods such as statistical relational learning. The theoretical and practical complexity of these approaches varies widely, so one of the primary purposes of a benchmark will be to quantify the trade-off between solution quality and runtime. We are motivated by the ubiquitous nature of entity resolution as a fundamental problem faced by any organization that ingests large amounts of noisy text data. A benchmark is typically a rigid specification that provides an objective measure usable for ranking implementations of an algorithm. For example the Top500 and HPCG500 bench- marks rank supercomputers based on their performance of dense and sparse linear algebra problems (respectively). These two benchmarks require participants to report FLOPS counts attainable on various machines. Our purpose is slightly different. Whereas the supercomputing benchmarks mentioned above hold algorithms constant and aim to rank machines, we are primarily interested in ranking algorithms. As mentioned above, entity resolution problems can be approached in completely different ways. We believe that users of our benchmarks must decide what sort of procedure to run before comparing implementations and architectures. Eventually, we also wish to provide a mechanism for ranking machines while holding algorithmic approach constant . Our primary contributions are parallel algorithms for computing solution quality mea- sures per entity. We find in some real datasets that many entities are quite easy to resolve while others are difficult, with a heavy skew toward the former case. Therefore, measures such as global confusion matrices, F measures, etc. do not meet our benchmarking needs. We design methods for computing solution quality at the granularity of a single entity in order to know when proposed solutions do well in difficult situations (perhaps justifying extra computational), or struggling in easy situations. We report on progress toward a viable benchmark for comparing entity resolution algo- rithms. Our work is incomplete, but we have designed and prototyped several algorithms to help evalute the solution quality of competing approaches to these problems. We envision a benchmark in which the objective measure is a ratio of solution quality to runtime.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Comparison Study of Upper Subcritical Limits Derived Using Sensitivity/Uncertainty Tools Case Studies of Benchmarks and Applications

Neutron transport methods used to establish subcriticality require validation by comparison to critical experiments considered to be benchmarks. Whisper is a sensitivity/uncertainty analysis tool developed to assist with the task of validation in nuclear criticality safety. Details on the Whisper methodology can be found in References 1-3 on the MCNP ® reference collection website at https://mcnp.lanl.gov. Whisper-1.0 was originally developed in 2014 and used to assist with nuclear criticality safety validation at Los Alamos National Laboratory. Whisper was upgraded in 2016 to Whisper-1.1 and prepared for release with MCNP6.2. Whisper contains a library of over 1100 critical experiment benchmarks and quantifies neutronic similarity of an application to benchmarks in the library. Using highest similarity benchmarks, Whisper computes a calculational margin (CM) encompassing of the worst-case bias and bias uncertainty at a 99% confidence level for each application. In addition, portions of the margin of subcriticality (MOS) for nuclear data uncertainty and potential code errors are computed. The baseline upper subcritical limit (USL) computed by Whisper is comprised of the CM, MOS nuclear data , and MOS code errors . The Whisper baseline USL is absent a portion of the MOS due to the area of application, which is applied based upon judgment by the criticality safety analyst. An objective of this paper is to present the baseline USL, CM and portions of the MOS as computed by Whisper for comparison with similar sensitivity/uncertainty tools, such as those used by IRSN and ORNL. The initial comparison involves four critical experiment benchmarks: HEU-MET-FAST-013-001, HEU-SOLTHERM-001-008, PU-MET-FAST-022-001, AND PU-SOL-THERM-001-001, which have been: 1. modeled independently by LANL, IRSN, and ORNL based upon information provided in the ICSBEP Handbook, 2. are common in S/U libraries for LANL, IRSN, and ORNL, 3. span a range of energy spectrum and fissile material, and 4. taken as applications for the purposes of this study and therefore excluded from use as a benchmark for calculating the upper subcritical limit. Results presented in this paper have been computed using covariance data for all isotopes in ENDF/BVII.0 using a 44-group energy structure. Benchmarks in the Whisper library were run in MCNP6.2 using 100,000 neutrons per cycle, skipping 100 cycles for 500 active cycles. Reference 10 also compares the results for baseline USL with an order of magnitude greater neutrons, using the same total number of cycles with 1,000,000 neutrons per cycle. Subsequent to the results presented in Reference 10 changes were made to the benchmark library, as discussed in Reference 11. Newer results using the revised benchmark library are presented herein.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2021

Sensitivity and Uncertainty of the IFR-1 BISON Benchmark

The fuel performance code BISON is being used to evaluate metallic fuel for a new fast-spectrum test reactor called the Versatile Test Reactor, which is being considered by the US Department of Energy. To quantify the accuracy of BISON predictions, researchers at Oak Ridge National Laboratory have been developing a series of benchmarks based on legacy metallic fuel experiments. As part of this effort, the sensitivity of BISON predictions to variations in model inputs and the uncertainties associated with BISON predictions must be established. This report summarizes efforts to perform a comprehensive sensitivity analysis (SA) and uncertainty quantification (UQ) on a benchmark based on the IFR-1 experiment. For the SA, at least one input was chosen from every BISON model and physics module used in the benchmark. The inputs were varied individually in a series of BISON simulations. The resulting variations in benchmark predictions were normalized to calculate sensitivities. The strongest sensitivities were identified and used to inform input selections for the UQ. The UQ was performed using the Monte Carlo UQ method. A literature review was conducted to estimate uncertainty distributions for the selected inputs, and values were sampled randomly from each distribution in a series of BISON simulations. Variations in the benchmark predictions were used to estimate uncertainty distributions and confidence intervals. It was found that nearly 100% of benchmark predictions matched the corresponding legacy values within the confidence intervals. However, this is at least partially because of the wide confidence intervals associated with the benchmark predictions. The uncertainty contributions of assumptions in the benchmark, experimental uncertainties, and BISON models were quantified. Some analysis was performed to identify inputs that contributed to the uncertainties. Finally, recommendations are made for future benchmark development and future BISON development.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2021

LANL Critical Benchmark Comparison Study and Subsequent Revision for Cases Involving LEU and MIX

As part of an international collaboration within the DOE Nuclear Criticality Safety Program (NCSP), LANL is involved in a comparison study to quantify differences in k-effective results from neutron transport simulations of critical benchmark experiments. The DOE NCSP Mission and Vision details the activity in which the French Institut de Radioprotection et de Sûreté Nucléaire (IRSN) leads the study with LANL, ORNL, and LLNL to compare results of various neutron transport codes and nuclear data libraries for ICSBEP benchmarks held in common by the entities. The task statement from the DOE NCSP Five-Year Execution Plan is given as: "The proposal is for IRSN to lead a new intercomparison based on the MORET code with the latest JEFF-3.3 data and ENDF/B-VIII.0 data, when available, using their existing comprehensive selection of 2,714 benchmarks and collate their results together with those from LLNL (COG), LANL (MCNP) and ORNL (SCALE). Due to the large number of benchmarks involved, this effort is envisioned to take three years with an additional year for IRSN to complete a summary report. The benchmark development will be performed independently to minimize modeling errors through discovery and resolution of discrepant results. A summary report will be generated (led by IRSN) to document the results of this study." This report documents results obtained for revisions made to cases involving Intermediate Enriched Uranium (IEU) and a mixture of Pu and Uranium (MIX), with a focus on the changes made to LANL benchmarks modeled with MCNP6 using ENDF/B-VII.1 nuclear data that appeared to have discrepant results when compared with results of other codes. Feedback was used to pinpoint review of particular benchmark input files and to revise them when necessary. This report documents the results of review and revision of specific benchmarks highlighted as possibly discrepant in the comparison study. In addition, there is an effort tied to this work involving collaboration between LANL XCP and NCS Divisions in the development of a shared review/revision procedure and use of a new benchmark repository.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2019

Criticality Accident Alarm System Shielding Benchmark: Integral Experiment Request 498, Critical Engineering Decision 1 Report

This report supports Critical/Subcritical Experiment Decision 1 (CED-1) for Integral Experiment Request (IER) 498, a shielding benchmark produced using the Godiva IV critical assembly at the National Critical Experiments Research Center (NCERC). Shielding benchmarks are datasets collected to provide a comparison for computer models that simulate the effect of shielding on radiation transmission from a source to a detector. The purpose of a CED-1 report is to produce a conceptual design for an integral experiment and to identify the long lead-time items. The design presented in this report favors certain detailed design options such as material selections. However, alternative options can be selected as the project evolves. Dimensions are specified only as needed to evaluate feasibility and performance. Development of Criticality Accident Alarm System (CAAS) shielding benchmarks supports two main programmatic needs. The first need is to improve confidence in models of facilities that house nuclear material handling operations. Better models can support effective CAAS detector placement when criticality may result in dose, and it enables more accurate analysis of criticality accident impacts at various locations in a facility. A key quantity that can be verified is the secondary gamma production caused by neutron interactions with various materials. The second programmatic need supported by shielding benchmarks is the ongoing development of methods being integrated into nuclear simulation codes such as the Monte Carlo N-Particle (MCNP) transport code and the SCALE code system used in efforts funded by the Nuclear Criticality Safety Program (NCSP). Previous work performed under the International Criticality Safety Benchmark Experimental Program (ICSBEP) to produce shielding benchmark data is primarily limited to a set of measurements at the SILENE critical assembly at the Valduc facility in France. The experiment used an open configuration and a burst mode operation, which limited detector options to thermoluminescent dosimeters (TLDs) and activation foils. Because TLDs do not record gamma spectra, it was difficult to verify the gamma energy spectrum from the experiment. In this work, the Godiva IV assembly will be used because it is a fast system capable of producing a wide range of source strengths, including super-prompt critical bursts. The room return for the facility is expected to be substantial and challenging to characterize, so a more closed, shielded experimental configuration is considered. Furthermore, because the physics of radiation transport outside the assembly does not change between the burst mode and the delayed critical mode, it is anticipated that most measurements will be made with the Godiva IV assembly in delayed critical mode, operating within a range that will allow electronic counters to collect accurate data. Scoping calculations concluded that a workable room return shield is comprised of 40 cm of 5% borated polyethylene backed by 7 cm of lead. This configuration was shown to produce a ratio of signal–to–room return of over 100 for up to 2 mean free paths of shield sample material, and over 10 for up to 4 mean free paths of shield sample material, ensuring that useful results can be obtained. With no shielding sample in the room return shield, it was shown that the neutron and gamma spectra do not appreciably change with a change in room size. This scoping calculation does not preclude the use of tungsten powder in a binder, some other effective gamma shielding material, or a hybrid design. Dose rates for various source intensities at an unshielded detector are listed herein to inform detector selection during CED-2, and calibration chains are identified. Preliminary estimates of costs and schedules are outlined but remain to be verified. Values could double or triple as estimates become more accurate and are principally included here to demonstrate feasibility. Preliminary estimates indicate that the apparatus can be installed within a feasible schedule and budget. Two measurements per day are assumed. Producing a shielding benchmark at NCERC using the Godiva IV assembly has been shown to be feasible, with manageable uncertainties and risks. A clear, continuing need for shielding benchmark data remains present, and this report identifies and evaluates a method for creating integrated shielding benchmark datasets.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

FAIR Surrogate Benchmarks Supporting AI and Simulation Research (Final Report)

Computational Science is being revolutionized by integrating AI and simulation and, in particular, by deep learning surrogate models that can replace all or part of traditional large‐scale HPC computations. Such surrogates can achieve remarkable performance improvements, as much as several orders of magnitude, and save both compute time and energy. The Surrogate Benchmark Initiative (SBI) project creates a community repository and FAIR (Findable, Accessible, Interoperable, and Reusable) data ecosystem for HPC application surrogate benchmarks. The SBI team comes from Argonne National Laboratory (ANL), Indiana University (IU), Rutgers University, the University of Tennessee, Knoxville (UTK), and the University of Virginia (UVA). SBI repositories include data, code, and all relevant collateral artifacts that the science and engineering community need to use and reuse these data sets and surrogates. SBI repositories generate active research from both the participants in SBI and the broad community of AI and domain scientists. This project develops surrogates that use several different neural nets to learn and quickly infer the results of simulations and data systems and captures them as surrogate benchmarks with a rich set of metadata covering: Data; Model; Metrics specification; Machine specification; and Science, Speed, and Power Results. We research FAIR metadata for these benchmarks. We develop application surrogate examples as benchmarks across many fields (ANL, UTK, IU, UVA). We also study non-Surrogate benchmarks that have many common features and similar issues as regards FAIRness. We work with MLCommons (UVA, UTK), which is a major machine learning benchmarking activity where we get metadata ontologies, software, and benchmarks, Benchmarks have datasets, models, and metadata and they need a technical framework developed by UTK and Rutgers and deployed by UVA. We study features of Surrogates including performance, training set size, and uncertainty quantification (Rutgers, UVA and IU).

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

A volumetric framework for quantum computer benchmarks

We propose a very large family of benchmarks for probing the performance of quantum computers. We call them volumetric benchmarks (VBs) because they generalize IBM's benchmark for measuring quantum volume \cite{Cross18}. The quantum volume benchmark defines a family of square circuits whose depth d and width w are the same. A volumetric benchmark defines a family of rectangular quantum circuits, for which d and w are uncoupled to allow the study of time/space performance trade-offs. Each VB defines a mapping from circuit shapes — ( w , d ) pairs — to test suites C ( w , d ) . A test suite is an ensemble of test circuits that share a common structure. The test suite C for a given circuit shape may be a single circuit C , a specific list of circuits { C 1 … C N } that must all be run, or a large set of possible circuits equipped with a distribution P r ( C ) . The circuits in a given VB share a structure, which is limited only by designers' creativity. We list some known benchmarks, and other circuit families, that fit into the VB framework: several families of random circuits, periodic circuits, and algorithm-inspired circuits. The last ingredient defining a benchmark is a success criterion that defines when a processor is judged to have ``passed'' a given test circuit. We discuss several options. Benchmark data can be analyzed in many ways to extract many properties, but we propose a simple, universal graphical summary of results that illustrates the Pareto frontier of the d vs w trade-off for the processor being benchmarked.

97 MATHEMATICS AND COMPUTING↗
NASA NTRSDate not supplied

Creating Benchmark Data for Artificial Intelligence and Machine Learning Space Biology Research

To identify an appropriate AI/ML approach for a specific problem, the best practice is to measure algorithm performance through the benchmarking process. A scientific benchmark consists of an AI-ready dataset and a reference implementation on a specific scientific question. The NASA Science Mission Directorate (SMD) has started the “Benchmark Initiative for AI/ML to create scientific benchmark datasets in three applications: 1) scientific benchmarking, which finds the best algorithm for a specific problem; 2) application benchmarking, which measures algorithm performance against a set of parameters; and 3) system benchmarking, which evaluates performance of hardware and software architecture. Currently, there are no standardized datasets available to benchmark AI/ML algorithms in the domain of space biology. In this work, we constructed two AI/ML-ready biological datasets from experiments in space-flown mice: cellular imaging and RNA-seq. First, radiation-exposed immune cells harbor DNA damage foci that can be fluorescently marked to visualize the amount of damage following exposure to ionizing radiation. However, such large datasets are difficult to analyze visually, due to imaging inconsistencies and human bias, and classical image processing approaches can fail on imaging artifacts. AI/ML are therefore exciting alternative, providing the speed of machines and the accuracy of humans. We have made this dataset available at https://registry.opendata.aws/bps_microscopy/. Second, high-throughput nucleic acid sequencing (DNA-seq, RNA-seq) has become widespread in biomedical research due to the growing availability and affordability of these assays. However, most sequencing datasets suffer from high dimensionality and low sample count. In this work, we used a generative adversarial network to synthesize a standardized, AI-ready, publicly available benchmark dataset for space biology RNA-seq data with sufficient space-flown and ground control mouse liver samples from NASA GeneLab. This dataset is available at https://registry.opendata.aws/bps_rnaseq/. These datasets are now fully open the Space Biology community to test their favorite AI/ML approaches.

James Casaletto↗