Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Criticality Accident Alarm System Shielding Benchmark: Integral Experiment Request 498, Critical Engineering Decision 1 Report

This report supports Critical/Subcritical Experiment Decision 1 (CED-1) for Integral Experiment Request (IER) 498, a shielding benchmark produced using the Godiva IV critical assembly at the National Critical Experiments Research Center (NCERC). Shielding benchmarks are datasets collected to provide a comparison for computer models that simulate the effect of shielding on radiation transmission from a source to a detector. The purpose of a CED-1 report is to produce a conceptual design for an integral experiment and to identify the long lead-time items. The design presented in this report favors certain detailed design options such as material selections. However, alternative options can be selected as the project evolves. Dimensions are specified only as needed to evaluate feasibility and performance. Development of Criticality Accident Alarm System (CAAS) shielding benchmarks supports two main programmatic needs. The first need is to improve confidence in models of facilities that house nuclear material handling operations. Better models can support effective CAAS detector placement when criticality may result in dose, and it enables more accurate analysis of criticality accident impacts at various locations in a facility. A key quantity that can be verified is the secondary gamma production caused by neutron interactions with various materials. The second programmatic need supported by shielding benchmarks is the ongoing development of methods being integrated into nuclear simulation codes such as the Monte Carlo N-Particle (MCNP) transport code and the SCALE code system used in efforts funded by the Nuclear Criticality Safety Program (NCSP). Previous work performed under the International Criticality Safety Benchmark Experimental Program (ICSBEP) to produce shielding benchmark data is primarily limited to a set of measurements at the SILENE critical assembly at the Valduc facility in France. The experiment used an open configuration and a burst mode operation, which limited detector options to thermoluminescent dosimeters (TLDs) and activation foils. Because TLDs do not record gamma spectra, it was difficult to verify the gamma energy spectrum from the experiment. In this work, the Godiva IV assembly will be used because it is a fast system capable of producing a wide range of source strengths, including super-prompt critical bursts. The room return for the facility is expected to be substantial and challenging to characterize, so a more closed, shielded experimental configuration is considered. Furthermore, because the physics of radiation transport outside the assembly does not change between the burst mode and the delayed critical mode, it is anticipated that most measurements will be made with the Godiva IV assembly in delayed critical mode, operating within a range that will allow electronic counters to collect accurate data. Scoping calculations concluded that a workable room return shield is comprised of 40 cm of 5% borated polyethylene backed by 7 cm of lead. This configuration was shown to produce a ratio of signal–to–room return of over 100 for up to 2 mean free paths of shield sample material, and over 10 for up to 4 mean free paths of shield sample material, ensuring that useful results can be obtained. With no shielding sample in the room return shield, it was shown that the neutron and gamma spectra do not appreciably change with a change in room size. This scoping calculation does not preclude the use of tungsten powder in a binder, some other effective gamma shielding material, or a hybrid design. Dose rates for various source intensities at an unshielded detector are listed herein to inform detector selection during CED-2, and calibration chains are identified. Preliminary estimates of costs and schedules are outlined but remain to be verified. Values could double or triple as estimates become more accurate and are principally included here to demonstrate feasibility. Preliminary estimates indicate that the apparatus can be installed within a feasible schedule and budget. Two measurements per day are assumed. Producing a shielding benchmark at NCERC using the Godiva IV assembly has been shown to be feasible, with manageable uncertainties and risks. A clear, continuing need for shielding benchmark data remains present, and this report identifies and evaluates a method for creating integrated shielding benchmark datasets.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

FAIR Surrogate Benchmarks Supporting AI and Simulation Research (Final Report)

Computational Science is being revolutionized by integrating AI and simulation and, in particular, by deep learning surrogate models that can replace all or part of traditional large‐scale HPC computations. Such surrogates can achieve remarkable performance improvements, as much as several orders of magnitude, and save both compute time and energy. The Surrogate Benchmark Initiative (SBI) project creates a community repository and FAIR (Findable, Accessible, Interoperable, and Reusable) data ecosystem for HPC application surrogate benchmarks. The SBI team comes from Argonne National Laboratory (ANL), Indiana University (IU), Rutgers University, the University of Tennessee, Knoxville (UTK), and the University of Virginia (UVA). SBI repositories include data, code, and all relevant collateral artifacts that the science and engineering community need to use and reuse these data sets and surrogates. SBI repositories generate active research from both the participants in SBI and the broad community of AI and domain scientists. This project develops surrogates that use several different neural nets to learn and quickly infer the results of simulations and data systems and captures them as surrogate benchmarks with a rich set of metadata covering: Data; Model; Metrics specification; Machine specification; and Science, Speed, and Power Results. We research FAIR metadata for these benchmarks. We develop application surrogate examples as benchmarks across many fields (ANL, UTK, IU, UVA). We also study non-Surrogate benchmarks that have many common features and similar issues as regards FAIRness. We work with MLCommons (UVA, UTK), which is a major machine learning benchmarking activity where we get metadata ontologies, software, and benchmarks, Benchmarks have datasets, models, and metadata and they need a technical framework developed by UTK and Rutgers and deployed by UVA. We study features of Surrogates including performance, training set size, and uncertainty quantification (Rutgers, UVA and IU).

97 MATHEMATICS AND COMPUTING↗

A volumetric framework for quantum computer benchmarks

We propose a very large family of benchmarks for probing the performance of quantum computers. We call them volumetric benchmarks (VBs) because they generalize IBM's benchmark for measuring quantum volume \cite{Cross18}. The quantum volume benchmark defines a family of square circuits whose depth d and width w are the same. A volumetric benchmark defines a family of rectangular quantum circuits, for which d and w are uncoupled to allow the study of time/space performance trade-offs. Each VB defines a mapping from circuit shapes — ( w , d ) pairs — to test suites C ( w , d ) . A test suite is an ensemble of test circuits that share a common structure. The test suite C for a given circuit shape may be a single circuit C , a specific list of circuits { C 1 … C N } that must all be run, or a large set of possible circuits equipped with a distribution P r ( C ) . The circuits in a given VB share a structure, which is limited only by designers' creativity. We list some known benchmarks, and other circuit families, that fit into the VB framework: several families of random circuits, periodic circuits, and algorithm-inspired circuits. The last ingredient defining a benchmark is a success criterion that defines when a processor is judged to have ``passed'' a given test circuit. We discuss several options. Benchmark data can be analyzed in many ways to extract many properties, but we propose a simple, universal graphical summary of results that illustrates the Pareto frontier of the d vs w trade-off for the processor being benchmarked.

97 MATHEMATICS AND COMPUTING↗

Recent Additions of Criticality Safety Related Integral Benchmark Data to the ICSBEP and IRPHEP Handbooks

High-quality integral benchmark experiments have always been a priority for criticality safety. However, interest in integral benchmark data is increasing as efforts to quantify and reduce calculational uncertainties accelerate to meet the demands of future criticality safety needs to support next generation reactor and advanced fuel cycle concepts. The importance of drawing upon existing benchmark data is becoming more apparent because of dwindling availability of critical facilities worldwide and the high cost of performing new experiments. Integral benchmark data from the International Handbook of Evaluated Criticality Safety Benchmark Experiments and the International Handbook of Reactor Physics Benchmark Experiments are widely used. Benchmark data have been added to these two handbooks since the last Nuclear Criticality Safety Division Topical Meeting in Knoxville, Tennessee (September 2005). This paper highlights these additions.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Nuclear Data Performance Testing Using Sensitive, but Less Frequently Used ICSBEP Benchmarks

The International Criticality Safety Benchmark Evaluation Project (ICSBEP) has published the International Handbook of Evaluated Criticality Safety Benchmark Experiments annually since 1995. The Handbook now spans over 51,000 pages with benchmark specifications for 4,283 critical, near critical, or subcritical configurations; 24 criticality alarm placement/shielding configurations with multiple dose points for each; and 200 configurations that have been categorized as fundamental physics measurements relevant to criticality safety applications. Benchmark data in the ICSBEP Handbook were originally intended for validation of criticality safety methods and data; however, the benchmark specifications are now used extensively for nuclear data testing. There are several, less frequently used benchmarks within the Handbook that are very sensitive to thorium and certain key structural and moderating materials. Calculated results for many of those benchmarks using modern nuclear data libraries suggest there is still room for improvement. These and other highly sensitive, but rarely quoted benchmarks are highlighted and data testing results provided using the Monte Carlo N-Particle Version 5 (MCNP5) code and continuous energy ENDF/B-V, VI.8, and VII.0, JEFF-3.1, and JENDL-3.3 nuclear data libraries.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Benchmark Evaluation of the NRAD Reactor LEU Core Startup Measurements

The Neutron Radiography (NRAD) reactor is a 250-kW TRIGA-(Training, Research, Isotope Production, General Atomics)-conversion-type reactor at the Idaho National Laboratory; it is primarily used for neutron radiography analysis of irradiated and unirradiated fuels and materials. The NRAD reactor was converted from HEU to LEU fuel with 60 fuel elements and brought critical on March 31, 2010. This configuration of the NRAD reactor has been evaluated as an acceptable benchmark experiment and is available in the 2011 editions of the International Handbook of Evaluated Criticality Safety Benchmark Experiments (ICSBEP Handbook) and the International Handbook of Evaluated Reactor Physics Benchmark Experiments (IRPhEP Handbook). Significant effort went into precisely characterizing all aspects of the reactor core dimensions and material properties; detailed analyses of reactor parameters minimized experimental uncertainties. The largest contributors to the total benchmark uncertainty were the 234 U, 236 U, Er, and Hf content in the fuel; the manganese content in the stainless-steel cladding; and the unknown level of water saturation in the graphite reflector blocks. A simplified benchmark model of the NRAD reactor was prepared with a keff of 1.0012 ± 0.0029 (1σ). Monte Carlo calculations with MCNP5 and KENO-VI and various neutron cross section libraries were performed and compared with the benchmark eigenvalue for the 60-fuel-element core configuration; all calculated eigenvalues are between 0.3 and 0.8% greater than the benchmark value. Benchmark evaluations of the NRAD reactor are beneficial in understanding biases and uncertainties affecting criticality safety analyses of storage, handling, or transportation applications with LEU-Er-Zr-H fuel.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Benchmark problems for numerical implementations of phase field models

Here, we present the first set of benchmark problems for phase field models that are being developed by the Center for Hierarchical Materials Design (CHiMaD) and the National Institute of Standards and Technology (NIST). While many scientific research areas use a limited set of well-established software, the growing phase field community continues to develop a wide variety of codes and lacks benchmark problems to consistently evaluate the numerical performance of new implementations. Phase field modeling has become significantly more popular as computational power has increased and is now becoming mainstream, driving the need for benchmark problems to validate and verify new implementations. We follow the example set by the micromagnetics community to develop an evolving set of benchmark problems that test the usability, computational resources, numerical capabilities and physical scope of phase field simulation codes. In this paper, we propose two benchmark problems that cover the physics of solute diffusion and growth and coarsening of a second phase via a simple spinodal decomposition model and a more complex Ostwald ripening model. We demonstrate the utility of benchmark problems by comparing the results of simulations performed with two different adaptive time stepping techniques, and we discuss the needs of future benchmark problems. The development of benchmark problems will enable the results of quantitative phase field models to be confidently incorporated into integrated computational materials science and engineering (ICME), an important goal of the Materials Genome Initiative.

36 MATERIALS SCIENCE↗

Estimating the Need for Palliative Radiation Therapy: A Benchmarking Approach

Purpose: Palliative radiation therapy (PRT) benefits many patients with incurable cancer, but the overall need for PRT is unknown. Our primary objective was to estimate the appropriate rate of use of PRT in Ontario. Methods and Materials: The Ontario Cancer Registry identified patients who died of cancer in Ontario between 2006 and 2010. Comprehensive RT records were linked to the registry. Multivariate analysis identified social and health system-related factors affecting the use of PRT, enabling us to define a benchmark population of patients with unimpeded access to PRT. The proportion of cases treated at any time (PRT{sub lifetime}), the proportion of cases treated in the last 2 years of life (PRT{sub 2y}), and number of courses of PRT per thousand cancer deaths were measured in the benchmark population. These benchmarks were standardized to the characteristics of the overall population, and province-wide PRT rates were then compared to benchmarks. Results: Cases diagnosed at hospitals with no RT on-site and residents of poorer communities and those who lived farther from an RT center, were significantly less likely than others to receive PRT. However, availability of RT at the diagnosing hospital was the dominant factor. Neither socioeconomic status nor distance from home to nearest RT center had a significant effect on the use of PRT in patients diagnosed at a hospital with RT facilities. The benchmark population therefore consisted of patients diagnosed at a hospital with RT facilities. The standardized benchmark for PRT{sub lifetime} was 33.9%, and the corresponding province-wide rate was 28.5%. The standardized benchmark for PRT{sub 2y} was 32.4%, and the corresponding province-wide rate was 27.0%. The standardized benchmark for the number of courses of PRT per thousand cancer deaths was 652, and the corresponding province-wide rate was 542. Conclusions: Approximately one-third of patients who die of cancer in Ontario need PRT, but many of them are never treated.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Simulation benchmarks for low-pressure plasmas: Capacitive discharges

Benchmarking is generally accepted as an important element in demonstrating the correctness of computer simulations. In the modern sense, a benchmark is a computer simulation result that has evidence of correctness, is accompanied by estimates of relevant errors, and which can thus be used as a basis for judging the accuracy and efficiency of other codes. In this paper, we present four benchmark cases related to capacitively coupled discharges. These benchmarks prescribe all relevant physical and numerical parameters. We have simulated the benchmark conditions using five independently developed particle-in-cell codes. We show that the results of these simulations are statistically indistinguishable, within bounds of uncertainty that we define. We, therefore, claim that the results of these simulations represent strong benchmarks, which can be used as a basis for evaluating the accuracy of other codes. These other codes could include other approaches than particle-in-cell simulations, where benchmarking could examine not just implementation accuracy and efficiency, but also the fidelity of different physical models, such as moment or hybrid models. We discuss an example of this kind in the Appendix. Of course, the methodology that we have developed can also be readily extended to a suite of benchmarks with coverage of a wider range of physical and chemical phenomena.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Benchmark Evaluation of HTR-PROTEUS Pebble Bed Experimental Program

Benchmark models were developed to evaluate 11 critical core configurations of the HTR-PROTEUS pebble bed experimental program. Various additional reactor physics measurements were performed as part of this program; currently only a total of 37 absorber rod worth measurements have been evaluated as acceptable benchmark experiments for Cores 4, 9, and 10. Dominant uncertainties in the experimental keff for all core configurations come from uncertainties in the ²³⁵U enrichment of the fuel, impurities in the moderator pebbles, and the density and impurity content of the radial reflector. Calculations of k eff with MCNP5 and ENDF/B-VII.0 neutron nuclear data are greater than the benchmark values but within 1% and also within the 3σ uncertainty, except for Core 4, which is the only randomly packed pebble configuration. Repeated calculations of k eff with MCNP6.1 and ENDF/B-VII.1 are lower than the benchmark values and within 1% (~3σ) except for Cores 5 and 9, which calculate lower than the benchmark eigenvalues within 4σ. The primary difference between the two nuclear data libraries is the adjustment of the absorption cross section of graphite. Simulations of the absorber rod worth measurements are within 3σ of the benchmark experiment values. The complete benchmark evaluation details are available in the 2014 edition of the International Handbook of Evaluated Reactor Physics Benchmark Experiments.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Gaussian Process Optimization of Sensitivity-Based Similarity Metrics between New Nuclear Applications and New/Existing Benchmarks [Slides]

This presentation discusses Nuclear Criticality Safety (NCS) and how designing safe, new nuclear criticality experiments requires expert judgement, which could take years of experience. Sensitivity/uncertainty (S/U) analysis can be utilized by less experienced individuals to conservatively estimate uncertainties in important parameters, such as k eff , in newly proposed nuclear experiments. The presentation poses the question of how this analysis can be performed and states that the answer lies in matching new nuclear experiments with existing benchmark experiments using similarity metrics. By increasing the criticality safety of the application in this work, higher mass limits could be used in PF-4 operations. Additionally, the presentation discusses MCNP6.2®, Whisper-1.1, the software that can be used in this analysis. Also discussed is the fact that International Criticality Safety Benchmark Evaluation Project (ICSBEP) benchmarks rarely match new nuclear applications and that there are significant differences in given set of materials and/or geometry. If there are no benchmarks that match the application, the presentation discusses the possibility of creating new benchmarks. In conclusion, this work presents a Gaussian process (GP) optimization scheme that was used to generate new benchmarks with the highest sensitivity-based similarity metrics to user-defined nuclear applications. The Gaussian process optimization successfully designed 3 new experimental benchmarks that were highly correlated to the application of interest and had k eff values near critical. Optimization over c k,i-r has shown that investigating specific isotope reactions for different applications is crucial to designing benchmark experiments. Partial contribution from Pu dominates c k similarity metric. Future work includes testing new stand-alone similarity metrics or new combinations of similarity metrics as the design criterion of this optimization – design criterion is application dependent.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

EUI Benchmarks for Net-Zero Energy Buildings in India

In our study we present EUI benchmarks for NZEBs for six building types across residential and non-residential typologies and for India's five climate zones. This approach is similar to the simulation-based benchmarks used by the B3 program in Minnesota, the Cal-Arch methodology in California, and the US Solar Decathlon approach, which combine simulations with actual building data. Of the six building types we explore one in detail with a range of operation scenarios, specifically narrowing down the EUI benchmarks for mixed-mode building operation and for variable temperature set-points as prescribed in the National Building Code of India.The contribution of this work is to provide rigorous end-use level EUI benchmarks for six building types, and to describe a method for simulation-based EUI benchmarks for mixed-mode operation with variable setpoints to highlight the difference between the standard approach used for the five building types and the On-Site Construction Worker Housing which additionally has the mixed-mode variable setpoint approach.On-Site Construction Worker Housings are typically poorly constructed temporary structures, without adequate thermal comfort, It is critical to provide adequate thermal comfort to protect people from the warming effects of climate change, and to discover super efficient and cost-effective ways to do so.The methodology used for the On-Site Construction Worker Housing (CWH) results in an 80% acceptability according to the India Model for Adaptive Comfort in the National Building Code of India. EUIs of all six buildings are 60% lower than the minimum compliance with India's Energy Conservation Building Codes, providing benchmarks for efficiency levels. The end-use level EUI benchmarks are now provided to over 1800 Solar Decathlon India participants so that they can compare the performance of their NZEB designs.In particular, the CWH results provide an insight into the importance of the mixed-mode operation with variable temperature set-points. The results from the simulation study show that for an NZEB target, the EUI with standard thermal comfort model and without mixed operation is 58.26 kWh/m2*year, while that with the variable set-points of the adaptive model with mixed mode operation is 24.23 kWh/m2*year. This is a 58% reduction in EUI. Given that many building types including residential, and non-residential operate in mixed mode, it is important to take this work further to develop mixed-mode operation NZEB benchmarks so that the carbon intensity of these buildings could be lower than the standard thermal comfort model approach.

benchmarking↗

Opportunities for enhancing MLCommons efforts while leveraging insights from educational MLCommons earthquake benchmarks efforts

MLCommons is an effort to develop and improve the artificial intelligence (AI) ecosystem through benchmarks, public data sets, and research. It consists of members from start-ups, leading companies, academics, and non-profits from around the world. The goal is to make machine learning better for everyone. In order to increase participation by others, educational institutions provide valuable opportunities for engagement. In this article, we identify numerous insights obtained from different viewpoints as part of efforts to utilize high-performance computing (HPC) big data systems in existing education while developing and conducting science benchmarks for earthquake prediction. As this activity was conducted across multiple educational efforts, we project if and how it is possible to make such efforts available on a wider scale. This includes the integration of sophisticated benchmarks into courses and research activities at universities, exposing the students and researchers to topics that are otherwise typically not sufficiently covered in current course curricula as we witnessed from our practical experience across multiple organizations. As such, we have outlined the many lessons we learned throughout these efforts, culminating in the need for benchmark carpentry for scientists using advanced computational resources. The article also presents the analysis of an earthquake prediction code benchmark while focusing on the accuracy of the results and not only on the runtime; notedly, this benchmark was created as a result of our lessons learned. Energy traces were produced throughout these benchmarks, which are vital to analyzing the power expenditure within HPC environments. Additionally, one of the insights is that in the short time of the project with limited student availability, the activity was only possible by utilizing a benchmark runtime pipeline while developing and using software to generate jobs from the permutation of hyperparameters automatically. It integrates a templated job management framework for executing tasks and experiments based on hyperparameters while leveraging hybrid compute resources available at different institutions. The software is part of a collection called cloudmesh with its newly developed components, cloudmesh-ee (experiment executor) and cloudmesh-cc (compute coordinator).

58 GEOSCIENCES↗

The ORSphere Benchmark Evaluation and Its Potential Impact on Nuclear Criticality Safety

In the early 1970’s, critical experiments using an unreflected metal sphere of highly enriched uranium (HEU) were performed with the focus to provide a "very accurate description…as an ideal benchmark for calculational methods and cross-section data files." Two near-critical configurations of the Oak Ridge Sphere (ORSphere) were evaluated as acceptable benchmark experiments for inclusion in the International Handbook of Evaluated Criticality Safety Benchmark Experiments (ICSBEP Handbook). The results from those benchmark experiments were then compared with additional unmoderated and unreflected HEU metal benchmark experiment configurations currently found in the ICSBEP Handbook. For basic geometries (spheres, cylinders, and slabs) the eigenvalues calculated using MCNP5 and ENDF/B-VII.0 were within 3 of their respective benchmark values. There appears to be generally a good agreement between calculated and benchmark values for spherical and slab geometry systems. Cylindrical geometry configurations tended to calculate low, including more complex bare HEU metal systems containing cylinders. The ORSphere experiments do not calculate within their 1σ uncertainty and there is a possibility that the effect of the measured uncertainties for the GODIVA I benchmark may need reevaluated. There is significant scatter in the calculations for the highly correlated ORCEF cylinder experiments, which are constructed from close-fitting HEU discs and annuli. Selection of a nuclear data library can have a larger impact on calculated eigenvalue results than the variation found within calculations of a given experimental series, such as the ORCEF cylinders, using a single nuclear data set.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Availability of Shielding Benchmark Experiment Data in the ICSBEP Handbook

The International Criticality Safety Benchmark Evaluation Projects (ICSBEP) has over two decades of experience under the auspices of the Organisation for Economic Co-operation and Development (OECD) Nuclear Energy Agency (NEA) to identify, evaluate, and compile a comprehensive data set of criticality safety benchmark data [1]. The activities of the ICSBEP have led to the continued success of the International Handbook of Evaluated Criticality Safety Benchmark Experiments (ICSBEP Handbook) [2]. The contents of the ICSBEP Handbook include benchmark specifications derived from nuclear facilities around the world, which have been utilized to support criticality safety efforts and also integral testing of nuclear data [3]. The 2020 edition of the ICSBEP Handbook will include data from 582 evaluations containing benchmark specifications for 5,053 critical, subcritical, or near-critical configurations. There are seven criticality-alarm-placement/shielding evaluations containing a total of 45 benchmark configurations, and ten fundamental physics evaluations containing a total of 237 measurements relevant to criticality safety applications. The purpose of this paper is to summarize those benchmarks relative to validation needs in shielding applications currently found within the ICSBEP Handbook. The current and future plans of shielding benchmarks with regards to the ICSBEP will also be discussed.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

New Version of the MCNP Analytic Criticality Benchmark Suite

Analytical benchmarks provide an invaluable tool for verifying computer codes used to simulate neutron transport. Several collections of analytical benchmark problems are used routinely in the verification of production Monte Carlo codes such as MCNP{sup R}. Verification of a computer code is a necessary prerequisite to the more complex validation process. The verification process confirms that a code performs its intended functions correctly. The validation process involves determining the absolute accuracy of code results vs. nature. In typical validations, results are computed for a set of benchmark experiments using a particular methodology (code, cross-section data with uncertainties, and modeling) and compared to the measured results from the set of benchmark experiments. The validation process determines bias, bias uncertainty, and possibly additional margins. Verification is generally performed by the code developers, while validation is generally performed by code users for a particular application space. The VERIFICATION-KEFF suite of criticality problems was originally a set of 75 criticality problems found in the literature for which exact analytical solutions are available. Even though the spatial and energy detail is necessarily limited in analytical benchmarks, typically to a few regions or energy groups, the exact solutions obtained can be used to verify that the basic algorithms, mathematics, and methods used in complex production codes perform correctly. The present work has focused on revisiting this benchmark suite. A thorough review of the problems resulted in discarding some of them as not suitable for MCNP benchmarking. For the remaining problems, many of them were reformulated to permit execution in either multigroup mode or in the normal continuous-energy mode for MCNP. Execution of the benchmarks in continuous-energy mode provides a significant advance to MCNP verification methods. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

A Comparison of Monte Carlo Particle Transport Algorithms for an Interior Source Binary Stochastic Medium Benchmark Suite

Particle transport through binary stochastic mixtures has received considerable research attention in the last two decades. Zimmerman and Adams proposed a Monte Carlo algorithm (Algorithm A) that solves the Levermore-Pomraning equations and another Monte Carlo algorithm (Algorithm B) that should be more accurate as a result of improved local material realization modeling. Zimmerman and Adams numerically confirmed these aspects of the Monte Carlo algorithms by comparing the reflection and transmission values computed using these algorithms to a standard suite of planar geometry binary stochastic mixture benchmark transport solutions. The benchmark transport problems are driven by an isotropic angular flux incident on one boundary of a binary Markovian statistical planar geometry medium. In a recent paper, we extended the benchmark comparisons of these Monte Carlo algorithms to include the scalar flux distributions produced. This comparison is important, because as demonstrated, an approximate model that gives accurate reflection and transmission probabilities can produce unphysical scalar flux distributions. Brantley and Palmer recently investigated the accuracy of the Levermore-Pomraning model using a new interior source binary stochastic medium benchmark problem suite. In this paper, we further investigate the accuracy of the Monte Carlo algorithms proposed by Zimmerman and Adams by comparing to the benchmark results from the interior source binary stochastic medium benchmark suite, including scalar flux distributions. Because the interior source scalar flux distributions are of an inherently different character than the distributions obtained for the incident angular flux benchmark problems, the present benchmark comparison extends the domain of problems for which the accuracy of these Monte Carlo algorithms has been investigated.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Impacts of benchmarking choices on inferred model skill of the Arctic–Boreal terrestrial carbon cycle

Abstract Land surface models require continuous validation against observations to improve and reduce simulation uncertainty. However, inferred model performance can be heavily influenced by subjective choices made in the selection and application of observational data products. A key area often misrepresented by models is the Arctic–Boreal region, which is a potential tipping point region in Earth’s climate system due to large permafrost carbon stocks that are vulnerable to release with climate warming. We use the International Land Model Benchmarking (ILAMB) framework to evaluate how the model skill of TRENDY-v9 models varies based on the choice of observational-based benchmark and how benchmarks are applied in model evaluation. This analysis uses global datasets integrated into ILAMB and new, regionally-specific observational products from the Arctic–Boreal Vulnerability Experiment. Our results cover the overall time period of 1979–2019 and show that model scores can vary substantially depending on the data product applied, with higher model scores indicating better model performance against observations. The lowest model scores occur when benchmarked against regional, compared to global, datasets. We also evaluate observed and modeled functional relationships between ecosystem respiration and air temperature and between gross primary production and precipitation. Here, we find that the magnitude and shape of the responses are strongly impacted by the choice of observational dataset and the approach used to construct the functional relationship benchmark. These results suggest that model evaluation studies could conclude a false sense of model skill if only using a single benchmark data product or if not applying regional data products when performing a regional model analysis. Collectively, our findings highlight the influence of benchmarking choices on model evaluation and point to the need for benchmarking guidelines when assessing model skill.

Poe, Jeralyn (ORCID:0000000318495278)↗