Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Preliminary Designs for Criticality Safety Benchmarks—Iron/Steel/Chromium Series

The nuclear criticality safety community uses critical benchmarks extensively when analyzing nuclear systems. Having benchmarks sensitive to important materials of interest helps to ensure safety and constrain results. Critical benchmarks are also invaluable to the nuclear data community, which uses these benchmarks as a check and measure of quality of nuclear data libraries. Additionally, integral benchmarks are often used to validate the performance of computational tools for simulating nuclear systems. Many integral benchmarks have been documented in the International Criticality Safety Benchmark Evaluation Project (ICSBEP) handbook, the International Reactor Physics Experiment Evaluation Project (IRPhEP) handbook, and the Shielding Integral Benchmark Archive Database (SINBAD). The National Criticality Experiments Research Center (NCERC), operated by Los Alamos National Laboratory, is the only general purpose critical experiments facility in the United States of America. NCERC regularly designs and executes critical experiments and other measurements useful to a wide variety of fields including nuclear criticality safety, commercial nuclear energy, nonproliferation, and nuclear data. NCERC has recently developed tools to visualize gaps and deficiencies in existing criticality benchmarks – this work is still ongoing. Through this work and other reviews, some high priority deficiencies have been identified. This paper aims to highlight three areas where additional critical benchmarks would be useful to the nuclear criticality safety community and various other groups. Additionally, NCERC is seeking feedback in regard to these proposed measurements to make sure they are as useful to as many communities as possible.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

SU-E-T-148: Benchmarks and Pre-Treatment Reviews: A Study of Quality Assurance Effectiveness

Purpose: To determine the impact benchmarks and pre-treatment reviews have on improving the quality of submitted clinical trial data. Methods: Benchmarks are used to evaluate a site’s ability to develop a treatment that meets a specific protocol’s treatment guidelines prior to placing their first patient on the protocol. A pre-treatment review is an actual patient placed on the protocol in which the dosimetry and contour volumes are evaluated to be per protocol guidelines prior to allowing the beginning of the treatment. A key component of these QA mechanisms is that sites are provided timely feedback to educate them on how to plan per the protocol and prevent protocol deviations on patients accrued to a protocol. For both benchmarks and pre-treatment reviews a dose volume analysis (DVA) was performed using MIM softwareTM. For pre-treatment reviews a volume contour evaluation was also performed. Results: IROC Houston performed a QA effectiveness analysis of a protocol which required both benchmarks and pre-treatment reviews. In 70 percent of the patient cases submitted, the benchmark played an effective role in assuring that the pre-treatment review of the cases met protocol requirements. The 35 percent of sites failing the benchmark subsequently modified there planning technique to pass the benchmark before being allowed to submit a patient for pre-treatment review. However, in 30 percent of the submitted cases the pre-treatment review failed where the majority (71 percent) failed the DVA. 20 percent of sites submitting patients failed to correct their dose volume discrepancies indicated by the benchmark case. Conclusion: Benchmark cases and pre-treatment reviews can be an effective QA tool to educate sites on protocol guidelines and to minimize deviations. Without the benchmark cases it is possible that 65 percent of the cases undergoing a pre-treatment review would have failed to meet the protocols requirements.Support: U24-CA-180803.

60 APPLIED LIFE SCIENCES↗

Benchmark Evaluation of Start-Up and Zero-Power Measurements at the High-Temperature Engineering Test Reactor

Benchmark models were developed to evaluate six cold-critical and two warm-critical, zero-power measurements of the HTTR. Additional measurements of a fully-loaded subcritical configuration, core excess reactivity, shutdown margins, six isothermal temperature coefficients, and axial reaction-rate distributions were also evaluated as acceptable benchmark experiments. Insufficient information is publicly available to develop finely-detailed models of the HTTR as much of the design information is still proprietary. However, the uncertainties in the benchmark models are judged to be of sufficient magnitude to encompass any biases and bias uncertainties incurred through the simplification process used to develop the benchmark models. Dominant uncertainties in the experimental keff for all core configurations come from uncertainties in the impurity content of the various graphite blocks that comprise the HTTR. Monte Carlo calculations of keff are between approximately 0.9 % and 2.7 % greater than the benchmark values. Reevaluation of the HTTR models as additional information becomes available could improve the quality of this benchmark and possibly reduce the computational biases. High-quality characterization of graphite impurities would significantly improve the quality of the HTTR benchmark assessment. Simulation of the other reactor physics measurements are in good agreement with the benchmark experiment values. The complete benchmark evaluation details are available in the 2014 edition of the International Handbook of Evaluated Reactor Physics Benchmark Experiments.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Guidebook for Using the Tool BEST Cement: Benchmarking and Energy Savings Tool for the Cement Industry

The Benchmarking and Energy Savings Tool (BEST) Cement is a process-based tool based on commercially available efficiency technologies used anywhere in the world applicable to the cement industry. This version has been designed for use in China. No actual cement facility with every single efficiency measure included in the benchmark will likely exist; however, the benchmark sets a reasonable standard by which to compare for plants striving to be the best. The energy consumption of the benchmark facility differs due to differences in processing at a given cement facility. The tool accounts for most of these variables and allows the user to adapt the model to operational variables specific for his/her cement facility. Figure 1 shows the boundaries included in a plant modeled by BEST Cement. In order to model the benchmark, i.e., the most energy efficient cement facility, so that it represents a facility similar to the user's cement facility, the user is first required to input production variables in the input sheet (see Section 6 for more information on how to input variables). These variables allow the tool to estimate a benchmark facility that is similar to the user's cement plant, giving a better picture of the potential for that particular facility, rather than benchmarking against a generic one. The input variables required include the following: (1) the amount of raw materials used in tonnes per year (limestone, gypsum, clay minerals, iron ore, blast furnace slag, fly ash, slag from other industries, natural pozzolans, limestone powder (used post-clinker stage), municipal wastes and others); the amount of raw materials that are preblended (prehomogenized and proportioned) and crushed (in tonnes per year); (2) the amount of additives that are dried and ground (in tonnes per year); (3) the production of clinker (in tonnes per year) from each kiln by kiln type; (4) the amount of raw materials, coal and clinker that is ground by mill type (in tonnes per year); (5) the amount of production of cement by type and grade (in tonnes per year); (6) the electricity generated onsite; and, (7) the energy used by fuel type; and, the amount (in RMB per year) spent on energy. The tool offers the user the opportunity to do a quick assessment or a more detailed assessment--this choice will determine the level of detail of the energy input. The detailed assessment will require energy data for each stage of production while the quick assessment will require only total energy used at the entire facility (see Section 6 for more details on quick versus detailed assessments). The benchmarking tool provides two benchmarks--one for Chinese best practices and one for international best practices. Section 2 describes the differences between these two and how each benchmark was calculated. The tool also asks for a target input by the user for the user to set goals for the facility.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Updated Godiva-IV Benchmark Preview

A note of errata prepended to the Godiva-IV delayed-critical benchmark (HEU-MET-FAST- 086) identifies two corrections to be made to the model: The glory hole in the spindle should be made larger, and the height of the safety block should be made smaller (and therefore its density made larger). In addition, the safety block at its full-in position is closer to the inner subassembly plate than was modeled in the benchmark. These changes have been made to HEU-MET-FAST-086 Case 4 in order to estimate the effect on $\kappa$ eff and on the neutron flux spectrum. Using smaller separation, a smaller safety block, and a larger glory hole caused keff to increase by 453 ± 1 pcm from the benchmark. The latest nuclear data, ENDF/B-VIII.0, have also been used, causing $\kappa$ eff to increase another 37 ± 1 pcm. Flux spectra were compared in a modeled fission foil (near the center of Godiva-IV in the glory hole) and at three external (point) detectors. Within the fission foil, using ENDF/B-VIII.0 induced changes in the flux spectrum similar in size to the changes due to using smaller separation, a smaller safety block, and a larger glory hole. At the detectors, using ENDF/B- VIII.0 induced changes in the flux spectrum much larger than those due to changing the model. In other words, the corrections to the benchmark model cause a large increase in $\kappa$ eff , but the changes to the neutron flux spectrum are small compared to those caused by using the latest nuclear data. This study presents a preview of results expected during the reevaluation of the Godiva IV benchmark, but it is not a substitute for the full reevaluation.A note of errata prepended to the Godiva-IV delayed-critical benchmark (HEU-MET-FAST- 086) identifies two corrections to be made to the model: The glory hole in the spindle should be made larger, and the height of the safety block should be made smaller (and therefore its density made larger). In addition, the safety block at its full-in position is closer to the inner subassembly plate than was modeled in the benchmark. These changes have been made to HEU-MET-FAST-086 Case 4 in order to estimate the effect on $\kappa$ eff and on the neutron flux spectrum. Using smaller separation, a smaller safety block, and a larger glory hole caused $\kappa$ eff to increase by 453 ± 1 pcm from the benchmark. The latest nuclear data, ENDF/B-VIII.0, have also been used, causing $\kappa$ eff to increase another 37 ± 1 pcm. Flux spectra were compared in a modeled fission foil (near the center of Godiva-IV in the glory hole) and at three external (point) detectors. Within the fission foil, using ENDF/B-VIII.0 induced changes in the flux spectrum similar in size to the changes due to using smaller separation, a smaller safety block, and a larger glory hole. At the detectors, using ENDF/B- VIII.0 induced changes in the flux spectrum much larger than those due to changing the model. In other words, the corrections to the benchmark model cause a large increase in $\kappa$ eff , but the changes to the neutron flux spectrum are small compared to those caused by using the latest nuclear data. This study presents a preview of results expected during the reevaluation of the Godiva IV benchmark, but it is not a substitute for the full reevaluation.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Validation and Independent Uncertainty Analysis of the MIX-SOL-THERM-003 ICSBEP Benchmark

The International Criticality Safety Benchmark Evaluation Project (ICSBEP) was started in 1992 by the United States Department of Energy and later in 1995 became an international project with contributions from 22 countries. The project is now organized by the OECD (Organisation for Economic Co-operation and Development) Nuclear Energy Agency (NEA). In its most recent iteration, the ICSBEP handbook contains over five thousand evaluations of critical, near-critical, and subcritical experiments conducted in facilities all around the world. These benchmarks serve as valuable information for criticality safety engineers who can use them to validate calculation techniques and establish minimum subcritical margins for operations with fissionable materials. The benchmarks in the handbook are categorized by their fissile material composition, material form (oxide, solution, or metal), and fission energy spectra. This is especially useful for those looking for benchmarks similar to a system they are working on to compare methods and identify trends. The ICSBEP Handbook Uncertainty Guide is document outlining recommended practices and methods for determining uncertainties in these benchmarks. Quantifying these uncertainties thoroughly is crucial as it allows a higher degree of confidence that data used from them is valid and relevant. The guide stresses the importance of a thorough and well documented uncertainty analysis when evaluating an experiment. All measured values of a system, whether they be dimensions or material compositions, have a certain amount of uncertainty associated with them and can be analyzed one by one to determine their effects on the system. Many evaluated benchmarks in the handbook present this in detail, however some do not, mostly earlier evaluations performed in the 1990’s and early 2000’s. Recently at Los Alamos National Laboratory (LANL), the Nuclear Criticality Safety Division (NCSD) of LANL has been validating MCNP6.2 ® input files of criticality benchmarks for use by Whisper, a criticality safety code developed at LANL. This effort is also part of the OECD NEA Working Party on International Nuclear Data Evaluation Co-operation (WPEC) Subgroup 45, also known as Validation of Nuclear Data Libraries (VaNDaL). The goal of VaNDaL is to compile a set of validated simulation inputs for use in validating nuclear data and simulation codes. As part of these efforts, one of the benchmarks reviewed was the MIX SOL-THERM-003 ICSBEP benchmark. This paper provides an independent uncertainty analysis of this benchmark experiment.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

U.S. Solar Photovoltaic System and Energy Storage Cost Benchmark: Q1 2021 [Slides]

Our benchmarking method includes bottom-up accounting for all necessary system and projectdevelopment costs incurred when installing residential, commercial, and utility-scale systems, and it models the Q1 2021 costs for such systems, excluding any previous supply agreements or contracts. In general, we attempt to model the typical installation techniques and business operations from an installed-cost perspective, and our benchmarks are national averages. The residential PV-only benchmark and the commercial rooftop PV-only benchmark average costs by inverter type (string inverters, string inverters with direct current [DC] optimizers, and microinverters), weighted by inverter market share. The residential PV-only benchmark is further averaged across small installer and national integrator business models, weighted by market share. All benchmarks include variations—accounting for the differences in size, equipment, and operational use (particularly for storage)—that are currently available in the marketplace. All benchmarks assume nonunionized construction labor; residential and commercial PV systems predominantly use nonunionized labor, and the type of labor required for utility-scale PV systems depends heavily on the development process. All benchmarks assume the use of monofacial monocrystalline silicon PV modules. Benchmarks using cadmium telluride or bifacial modules could result in significantly different results. The data in this annual benchmark report inform the formulation of and track progress toward the U.S. Department of Energy (DOE) Solar Energy Technologies Office’s Government Performance and Reporting Act cost targets.

14 SOLAR ENERGY↗

Summary of LANL Critical Benchmark Comparison Study and Revisions for Cases Involving HEU, LEU, MIX, and Pu

This report documents results obtained for revisions made to cases involving Highly Enriched Uranium (HEU), Intermediate Enriched Uranium (IEU), a mixture of Pu and Uranium (MIX), as well as Pu cases. A previous summary of revisions for HEU an Pu cases was reported and additional investigations into four cases originally presented therein uncovered further revisions which led to better agreement with other transport codes, those cases are updated in this report. The summary of all cases reported in Reference 2 is updated in this report. In addition, a previous summary of revisions for LEU and MIX was reported, a summary of those revisions in reproduced in this report for a comprehensive summary of changes to benchmarks beginning in fiscal year 2020 to current date. The report focuses on the changes made to LANL benchmarks modeled with MCNP6 using ENDF/B-VII.1 nuclear data that appeared to have discrepant results when compared with results of other codes. Feedback was used to pinpoint review of benchmark input files and to revise them when necessary. This report documents the results of review and revision of specific benchmarks highlighted as possibly discrepant in the comparison study. In addition, there is an effort tied to this work involving collaboration between LANL XCP and NCS Divisions in the development of a shared review/revision procedure and use of a new benchmark repository. LANL has a benchmark library of critical experiments from the International Criticality Safety Benchmark Evaluation Project (ICSBEP) Handbook modeled for use with MCNP. This collection is now over 1100 benchmarks, referred to as the Whisper-1.1 library because it is used with the sensitivity/uncertainty package, Whisper, which supports nuclear criticality safety validation and is released with MCNP6.2. The collection, originally created several decades ago, is a combination of smaller collections, which has been revised and expanded, by various groups at LANL over the years. The original authors are no longer at the laboratory and little formal documentation of review and revision of these benchmarks exists today. A branch of the benchmark collection was already the subject of a formal review undertaken by the LANL NCS Division and expanded to include XCP Division.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

An MLCommons Scientific Benchmarks Ontology

Scientific machine learning research spans diverse domains and data modalities, yet existing benchmark efforts remain siloed and lack standardization. This makes novel and transformative applications of machine learning to critical scientific use-cases more fragmented and less clear in pathways to impact. This paper introduces an ontology for scientific benchmarking developed through a unified, community-driven effort that extends the MLCommons ecosystem to cover physics, chemistry, materials science, biology, climate science, and more. Building on prior initiatives such as XAI-BENCH, FastML Science Benchmarks, PDEBench, and the SciMLBench framework, our effort consolidates a large set of disparate benchmarks and frameworks into a single taxonomy of scientific, application, and system-level benchmarks. New benchmarks can be added through an open submission workflow coordinated by the MLCommons Science Working Group and evaluated against a six-category rating rubric that promotes and identifies high-quality benchmarks, enabling stakeholders to select benchmarks that meet their specific needs. The architecture is extensible, supporting future scientific and AI/ML motifs, and we discuss methods for identifying emerging computing patterns for unique scientific workloads. The MLCommons Science Benchmarks Ontology provides a standardized, scalable foundation for reproducible, cross-domain benchmarking in scientific machine learning. A companion webpage for this work has also been developed as the effort evolves: https://mlcommons-science.github.io/benchmark/

Hawks, Ben [Fermilab] (ORCID:0000000157000288)↗

Gaussian Process Optimization of Sensitivity-Based Similarity Metrics between New Nuclear Applications and New/Existing Benchmarks

Confidently designing safe, new nuclear criticality experiments requires expert judgement, which could take years of experience. Sensitivity/uncertainty (S/U) analysis can be utilized by less experienced individuals to conservatively estimate uncertainties in important parameters, such as k eff , in newly proposed nuclear applications. This type of analysis relies on matching new nuclear applications with existing benchmark experiments. The Whisper-1.1 software package included in MCNP6.2 ®1 contains more than 1,100 International Criticality Safety Benchmark Evaluation Project (ICSBEP) benchmarks. These benchmarks however rarely match new nuclear applications. The number of benchmarks available to match a given set of materials or geometric configurations varies significantly. Furthermore, recently performed benchmark experiments may not have had enough time to be properly documented and published. Benchmarks are vital for determining the accuracy of nuclear data and can assist nuclear physics and evaluators in improving nuclear data libraries. Exhaustively exploring the parameter space using simulations with software such as MCNP is too computationally expensive. In this work, Gaussian process optimization was implemented to reduce the number of simulations needed for optimization over multiple parameters. This optimization scheme was designed to selectively generate new benchmarks with high sensitivity-based similarity metrics to user-defined nuclear applications. Two benchmark models of spherically nested shells containing plutonium, uranium, tantalum, and water were used in the optimization to match an application containing plutonium plates, stacked in a tantalum reflector, surrounded by water.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Sensitivity and uncertainty of the IFR-1 BISON benchmark

The fuel performance code BISON is being used to evaluate metallic fuel for a new fast-spectrum test reactor called the Versatile Test Reactor, which is being considered for adoption by the US Department of Energy. To quantify the accuracy of BISON predictions, researchers at Oak Ridge National Laboratory have been developing a series of benchmarks based on legacy metallic fuel experiments. As part of this effort, the sensitivity of BISON predictions to variations in model inputs and the uncertainties associated with BISON predictions must be established. This paper summarizes efforts to perform a comprehensive sensitivity analysis (SA) and uncertainty quantification (UQ) on a benchmark based on the IFR-1 experiment.For the SA, at least one input was chosen from every BISON model and physics module used in the benchmark. The inputs were varied individually in a series of BISON simulations. Here, the resulting variations in benchmark predictions were normalized to calculate sensitivities. These sensitivities were then used to inform input selections for the UQ.The UQ was performed using the Monte Carlo UQ method. A literature review was conducted to estimate uncertainty distributions for the selected inputs, and values were sampled randomly from each distribution in a series of BISON simulations. Variations in the benchmark predictions were used to estimate uncertainty distributions and confidence intervals. It was found that nearly 100% of the benchmark predictions matched the corresponding legacy values within the confidence intervals. However, this is at least partially because the confidence intervals associated with benchmark predictions were wide. The uncertainty contributions of assumptions in the benchmark, experimental uncertainties, and BISON models were quantified. Some analysis was performed to identify inputs that contributed to the uncertainties. Finally, recommendations are made for future benchmark and future BISON development.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Informing Robust Functional Relationship Benchmarks: An Evaluation of the Temperature Sensitivity of Ecosystem Respiration Across the Arctic-Boreal Region

During land model development, simulated carbon dynamics are often benchmarked against observational data sets to evaluate model performance. Functional relationship benchmarks are the relationship between a driving variable (e.g., temperature) and a response variable (e.g., ecosystem respiration) and are a promising tool for assessing model performance by evaluating modeled sensitivities to changing environmental conditions. However, observed functional relationships can be influenced by choices made during data collection and throughout the benchmarking process, impacting the inferred skill of land models. To avoid misrepresenting a model's true performance, it is necessary to systematically evaluate best practices when constructing functional relationship benchmarks. We developed a set of guidelines for constructing functional relationship benchmarks, considering the choice of data set, number of daily observations, temporal extent, and temporal resolution across Alaska and Canada over a 20-year period from 2001 to 2020. The temperature sensitivity of ecosystem respiration from observations, evaluated through an apparent Q 10 , is highly variable both spatially and as a result of the data processing approach applied in the benchmark formation. When benchmarking 13 models from the Warming Permafrost Model Intercomparison Project (WrPMIP), the range in inferred model skill is substantially impacted by the choices applied in constructing functional relationship benchmarks. The inferred performance of a given model is most sensitive to the number of daily observations and temporal extent, followed by choice of benchmark data set and temporal averaging. Results from this analysis can guide the development of consistent and robust functional relationships for future model evaluation studies.

Poe, Jeralyn [Northern Arizona University, Flagsta↗

Benchmarking quantum computers

The rapid pace of development in quantum computing technology has sparked a proliferation of benchmarks to assess the performance of quantum computing hardware and software. However, not all benchmarks are of equal merit. Good ones empower scientists, engineers, programmers and users to understand the power of a computing system, whereas bad ones can misdirect research and inhibit progress. In this Perspective, we survey the science of quantum computer benchmarking. Here, we discuss the role of benchmarks and benchmarking and how good benchmarks can drive and measure progress towards the long-term goal of useful quantum computations, known as quantum utility. We explain how different kinds of benchmark quantify the performance of different parts of a quantum computer, discuss existing benchmarks, examine recent trends in benchmarking, and highlight important open research questions in this field.

Proctor, Timothy James [Sandia National Laboratori↗

Benchmarking for AI for Science

AI has been instrumental for recent developments in a number of domains of the sciences. With several hundred machine learning (ML) algorithms and models, and numerous AI-specific hardware platforms, a common quest for all scientists working on AI for Science is around the selection of machine learning algorithm(s) to solve their domain-specific scientific problems. A number of different initiatives around AI Benchmarking have been set up and have been useful in understanding the benefits of different ML algorithms for different tasks.However, with the majority of these AI Benchmarking initiatives focusing on the conventional notions of benchmarking, where the focus is purely runtime performance (such as training time or inference time), their suitability for benchmarking different ML algorithms for solving scientific problems has been viewed as a performance problem even though both are hardly the same. To make reasonable, explainable, and justifiable advancements in science using AI, it is critical to focus on the merits of these algorithms in handling different domain science problems. In other words, more emphasis must be given on Benchmarking for AI for Science than AI Benchmarking. The vision of the former is not only to assess the performance of ML algorithms, but also to assess, and understand the benefits and merits of different ML algorithms in handling scientific problems. Benchmarking for AI for Science, instead of pure performance focused AI Benchmarking, has several benefits: (i) it has the potential to offer advances in the sciences, much more rapidly than through pure performance-based AI methods, (ii) it will encourage the community to focus on developing better domain-specific AI techniques, particularly given the provision for being able to benchmark different techniques, and (iii) it will encourage hardware manufacturers to focus on developing science-specific hardware subsystems.

Thiyagalingam, Jeyan↗

Entity Resolution at Large Scale: Benchmarking and Algorithmics

We seek scalable benchmarks for entity resolution problems. Solutions to these problems range from trivial approaches such as string sorting to sophisticated methods such as statistical relational learning. The theoretical and practical complexity of these approaches varies widely, so one of the primary purposes of a benchmark will be to quantify the trade-off between solution quality and runtime. We are motivated by the ubiquitous nature of entity resolution as a fundamental problem faced by any organization that ingests large amounts of noisy text data. A benchmark is typically a rigid specification that provides an objective measure usable for ranking implementations of an algorithm. For example the Top500 and HPCG500 bench- marks rank supercomputers based on their performance of dense and sparse linear algebra problems (respectively). These two benchmarks require participants to report FLOPS counts attainable on various machines. Our purpose is slightly different. Whereas the supercomputing benchmarks mentioned above hold algorithms constant and aim to rank machines, we are primarily interested in ranking algorithms. As mentioned above, entity resolution problems can be approached in completely different ways. We believe that users of our benchmarks must decide what sort of procedure to run before comparing implementations and architectures. Eventually, we also wish to provide a mechanism for ranking machines while holding algorithmic approach constant . Our primary contributions are parallel algorithms for computing solution quality mea- sures per entity. We find in some real datasets that many entities are quite easy to resolve while others are difficult, with a heavy skew toward the former case. Therefore, measures such as global confusion matrices, F measures, etc. do not meet our benchmarking needs. We design methods for computing solution quality at the granularity of a single entity in order to know when proposed solutions do well in difficult situations (perhaps justifying extra computational), or struggling in easy situations. We report on progress toward a viable benchmark for comparing entity resolution algo- rithms. Our work is incomplete, but we have designed and prototyped several algorithms to help evalute the solution quality of competing approaches to these problems. We envision a benchmark in which the objective measure is a ratio of solution quality to runtime.

97 MATHEMATICS AND COMPUTING↗

Comparison Study of Upper Subcritical Limits Derived Using Sensitivity/Uncertainty Tools Case Studies of Benchmarks and Applications

Neutron transport methods used to establish subcriticality require validation by comparison to critical experiments considered to be benchmarks. Whisper is a sensitivity/uncertainty analysis tool developed to assist with the task of validation in nuclear criticality safety. Details on the Whisper methodology can be found in References 1-3 on the MCNP ® reference collection website at https://mcnp.lanl.gov. Whisper-1.0 was originally developed in 2014 and used to assist with nuclear criticality safety validation at Los Alamos National Laboratory. Whisper was upgraded in 2016 to Whisper-1.1 and prepared for release with MCNP6.2. Whisper contains a library of over 1100 critical experiment benchmarks and quantifies neutronic similarity of an application to benchmarks in the library. Using highest similarity benchmarks, Whisper computes a calculational margin (CM) encompassing of the worst-case bias and bias uncertainty at a 99% confidence level for each application. In addition, portions of the margin of subcriticality (MOS) for nuclear data uncertainty and potential code errors are computed. The baseline upper subcritical limit (USL) computed by Whisper is comprised of the CM, MOS nuclear data , and MOS code errors . The Whisper baseline USL is absent a portion of the MOS due to the area of application, which is applied based upon judgment by the criticality safety analyst. An objective of this paper is to present the baseline USL, CM and portions of the MOS as computed by Whisper for comparison with similar sensitivity/uncertainty tools, such as those used by IRSN and ORNL. The initial comparison involves four critical experiment benchmarks: HEU-MET-FAST-013-001, HEU-SOLTHERM-001-008, PU-MET-FAST-022-001, AND PU-SOL-THERM-001-001, which have been: 1. modeled independently by LANL, IRSN, and ORNL based upon information provided in the ICSBEP Handbook, 2. are common in S/U libraries for LANL, IRSN, and ORNL, 3. span a range of energy spectrum and fissile material, and 4. taken as applications for the purposes of this study and therefore excluded from use as a benchmark for calculating the upper subcritical limit. Results presented in this paper have been computed using covariance data for all isotopes in ENDF/BVII.0 using a 44-group energy structure. Benchmarks in the Whisper library were run in MCNP6.2 using 100,000 neutrons per cycle, skipping 100 cycles for 500 active cycles. Reference 10 also compares the results for baseline USL with an order of magnitude greater neutrons, using the same total number of cycles with 1,000,000 neutrons per cycle. Subsequent to the results presented in Reference 10 changes were made to the benchmark library, as discussed in Reference 11. Newer results using the revised benchmark library are presented herein.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Sensitivity and Uncertainty of the IFR-1 BISON Benchmark

The fuel performance code BISON is being used to evaluate metallic fuel for a new fast-spectrum test reactor called the Versatile Test Reactor, which is being considered by the US Department of Energy. To quantify the accuracy of BISON predictions, researchers at Oak Ridge National Laboratory have been developing a series of benchmarks based on legacy metallic fuel experiments. As part of this effort, the sensitivity of BISON predictions to variations in model inputs and the uncertainties associated with BISON predictions must be established. This report summarizes efforts to perform a comprehensive sensitivity analysis (SA) and uncertainty quantification (UQ) on a benchmark based on the IFR-1 experiment. For the SA, at least one input was chosen from every BISON model and physics module used in the benchmark. The inputs were varied individually in a series of BISON simulations. The resulting variations in benchmark predictions were normalized to calculate sensitivities. The strongest sensitivities were identified and used to inform input selections for the UQ. The UQ was performed using the Monte Carlo UQ method. A literature review was conducted to estimate uncertainty distributions for the selected inputs, and values were sampled randomly from each distribution in a series of BISON simulations. Variations in the benchmark predictions were used to estimate uncertainty distributions and confidence intervals. It was found that nearly 100% of benchmark predictions matched the corresponding legacy values within the confidence intervals. However, this is at least partially because of the wide confidence intervals associated with the benchmark predictions. The uncertainty contributions of assumptions in the benchmark, experimental uncertainties, and BISON models were quantified. Some analysis was performed to identify inputs that contributed to the uncertainties. Finally, recommendations are made for future benchmark development and future BISON development.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

LANL Critical Benchmark Comparison Study and Subsequent Revision for Cases Involving LEU and MIX

As part of an international collaboration within the DOE Nuclear Criticality Safety Program (NCSP), LANL is involved in a comparison study to quantify differences in k-effective results from neutron transport simulations of critical benchmark experiments. The DOE NCSP Mission and Vision details the activity in which the French Institut de Radioprotection et de Sûreté Nucléaire (IRSN) leads the study with LANL, ORNL, and LLNL to compare results of various neutron transport codes and nuclear data libraries for ICSBEP benchmarks held in common by the entities. The task statement from the DOE NCSP Five-Year Execution Plan is given as: "The proposal is for IRSN to lead a new intercomparison based on the MORET code with the latest JEFF-3.3 data and ENDF/B-VIII.0 data, when available, using their existing comprehensive selection of 2,714 benchmarks and collate their results together with those from LLNL (COG), LANL (MCNP) and ORNL (SCALE). Due to the large number of benchmarks involved, this effort is envisioned to take three years with an additional year for IRSN to complete a summary report. The benchmark development will be performed independently to minimize modeling errors through discovery and resolution of discrepant results. A summary report will be generated (led by IRSN) to document the results of this study." This report documents results obtained for revisions made to cases involving Intermediate Enriched Uranium (IEU) and a mixture of Pu and Uranium (MIX), with a focus on the changes made to LANL benchmarks modeled with MCNP6 using ENDF/B-VII.1 nuclear data that appeared to have discrepant results when compared with results of other codes. Feedback was used to pinpoint review of particular benchmark input files and to revise them when necessary. This report documents the results of review and revision of specific benchmarks highlighted as possibly discrepant in the comparison study. In addition, there is an effort tied to this work involving collaboration between LANL XCP and NCS Divisions in the development of a shared review/revision procedure and use of a new benchmark repository.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗