Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “BENCHMARKS”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

A volumetric framework for quantum computer benchmarks

We propose a very large family of benchmarks for probing the performance of quantum computers. We call them volumetric benchmarks (VBs) because they generalize IBM's benchmark for measuring quantum volume \cite{Cross18}. The quantum volume benchmark defines a family of square circuits whose depth d and width w are the same. A volumetric benchmark defines a family of rectangular quantum circuits, for which d and w are uncoupled to allow the study of time/space performance trade-offs. Each VB defines a mapping from circuit shapes — ( w , d ) pairs — to test suites C ( w , d ) . A test suite is an ensemble of test circuits that share a common structure. The test suite C for a given circuit shape may be a single circuit C , a specific list of circuits { C 1 … C N } that must all be run, or a large set of possible circuits equipped with a distribution P r ( C ) . The circuits in a given VB share a structure, which is limited only by designers' creativity. We list some known benchmarks, and other circuit families, that fit into the VB framework: several families of random circuits, periodic circuits, and algorithm-inspired circuits. The last ingredient defining a benchmark is a success criterion that defines when a processor is judged to have ``passed'' a given test circuit. We discuss several options. Benchmark data can be analyzed in many ways to extract many properties, but we propose a simple, universal graphical summary of results that illustrates the Pareto frontier of the d vs w trade-off for the processor being benchmarked.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Gaussian Process Optimization of Sensitivity-Based Similarity Metrics between New Nuclear Applications and New/Existing Benchmarks [Slides]

This presentation discusses Nuclear Criticality Safety (NCS) and how designing safe, new nuclear criticality experiments requires expert judgement, which could take years of experience. Sensitivity/uncertainty (S/U) analysis can be utilized by less experienced individuals to conservatively estimate uncertainties in important parameters, such as k eff , in newly proposed nuclear experiments. The presentation poses the question of how this analysis can be performed and states that the answer lies in matching new nuclear experiments with existing benchmark experiments using similarity metrics. By increasing the criticality safety of the application in this work, higher mass limits could be used in PF-4 operations. Additionally, the presentation discusses MCNP6.2®, Whisper-1.1, the software that can be used in this analysis. Also discussed is the fact that International Criticality Safety Benchmark Evaluation Project (ICSBEP) benchmarks rarely match new nuclear applications and that there are significant differences in given set of materials and/or geometry. If there are no benchmarks that match the application, the presentation discusses the possibility of creating new benchmarks. In conclusion, this work presents a Gaussian process (GP) optimization scheme that was used to generate new benchmarks with the highest sensitivity-based similarity metrics to user-defined nuclear applications. The Gaussian process optimization successfully designed 3 new experimental benchmarks that were highly correlated to the application of interest and had k eff values near critical. Optimization over c k,i-r has shown that investigating specific isotope reactions for different applications is crucial to designing benchmark experiments. Partial contribution from Pu dominates c k similarity metric. Future work includes testing new stand-alone similarity metrics or new combinations of similarity metrics as the design criterion of this optimization – design criterion is application dependent.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2023

EUI Benchmarks for Net-Zero Energy Buildings in India

In our study we present EUI benchmarks for NZEBs for six building types across residential and non-residential typologies and for India's five climate zones. This approach is similar to the simulation-based benchmarks used by the B3 program in Minnesota, the Cal-Arch methodology in California, and the US Solar Decathlon approach, which combine simulations with actual building data. Of the six building types we explore one in detail with a range of operation scenarios, specifically narrowing down the EUI benchmarks for mixed-mode building operation and for variable temperature set-points as prescribed in the National Building Code of India.The contribution of this work is to provide rigorous end-use level EUI benchmarks for six building types, and to describe a method for simulation-based EUI benchmarks for mixed-mode operation with variable setpoints to highlight the difference between the standard approach used for the five building types and the On-Site Construction Worker Housing which additionally has the mixed-mode variable setpoint approach.On-Site Construction Worker Housings are typically poorly constructed temporary structures, without adequate thermal comfort, It is critical to provide adequate thermal comfort to protect people from the warming effects of climate change, and to discover super efficient and cost-effective ways to do so.The methodology used for the On-Site Construction Worker Housing (CWH) results in an 80% acceptability according to the India Model for Adaptive Comfort in the National Building Code of India. EUIs of all six buildings are 60% lower than the minimum compliance with India's Energy Conservation Building Codes, providing benchmarks for efficiency levels. The end-use level EUI benchmarks are now provided to over 1800 Solar Decathlon India participants so that they can compare the performance of their NZEB designs.In particular, the CWH results provide an insight into the importance of the mixed-mode operation with variable temperature set-points. The results from the simulation study show that for an NZEB target, the EUI with standard thermal comfort model and without mixed operation is 58.26 kWh/m2*year, while that with the variable set-points of the adaptive model with mixed mode operation is 24.23 kWh/m2*year. This is a 58% reduction in EUI. Given that many building types including residential, and non-residential operate in mixed mode, it is important to take this work further to develop mixed-mode operation NZEB benchmarks so that the carbon intensity of these buildings could be lower than the standard thermal comfort model approach.

benchmarking↗
DOE OSTI2023

Opportunities for enhancing MLCommons efforts while leveraging insights from educational MLCommons earthquake benchmarks efforts

MLCommons is an effort to develop and improve the artificial intelligence (AI) ecosystem through benchmarks, public data sets, and research. It consists of members from start-ups, leading companies, academics, and non-profits from around the world. The goal is to make machine learning better for everyone. In order to increase participation by others, educational institutions provide valuable opportunities for engagement. In this article, we identify numerous insights obtained from different viewpoints as part of efforts to utilize high-performance computing (HPC) big data systems in existing education while developing and conducting science benchmarks for earthquake prediction. As this activity was conducted across multiple educational efforts, we project if and how it is possible to make such efforts available on a wider scale. This includes the integration of sophisticated benchmarks into courses and research activities at universities, exposing the students and researchers to topics that are otherwise typically not sufficiently covered in current course curricula as we witnessed from our practical experience across multiple organizations. As such, we have outlined the many lessons we learned throughout these efforts, culminating in the need for benchmark carpentry for scientists using advanced computational resources. The article also presents the analysis of an earthquake prediction code benchmark while focusing on the accuracy of the results and not only on the runtime; notedly, this benchmark was created as a result of our lessons learned. Energy traces were produced throughout these benchmarks, which are vital to analyzing the power expenditure within HPC environments. Additionally, one of the insights is that in the short time of the project with limited student availability, the activity was only possible by utilizing a benchmark runtime pipeline while developing and using software to generate jobs from the permutation of hyperparameters automatically. It integrates a templated job management framework for executing tasks and experiments based on hyperparameters while leveraging hybrid compute resources available at different institutions. The software is part of a collection called cloudmesh with its newly developed components, cloudmesh-ee (experiment executor) and cloudmesh-cc (compute coordinator).

58 GEOSCIENCES↗
DOE OSTI2020

Availability of Shielding Benchmark Experiment Data in the ICSBEP Handbook

The International Criticality Safety Benchmark Evaluation Projects (ICSBEP) has over two decades of experience under the auspices of the Organisation for Economic Co-operation and Development (OECD) Nuclear Energy Agency (NEA) to identify, evaluate, and compile a comprehensive data set of criticality safety benchmark data [1]. The activities of the ICSBEP have led to the continued success of the International Handbook of Evaluated Criticality Safety Benchmark Experiments (ICSBEP Handbook) [2]. The contents of the ICSBEP Handbook include benchmark specifications derived from nuclear facilities around the world, which have been utilized to support criticality safety efforts and also integral testing of nuclear data [3]. The 2020 edition of the ICSBEP Handbook will include data from 582 evaluations containing benchmark specifications for 5,053 critical, subcritical, or near-critical configurations. There are seven criticality-alarm-placement/shielding evaluations containing a total of 45 benchmark configurations, and ten fundamental physics evaluations containing a total of 237 measurements relevant to criticality safety applications. The purpose of this paper is to summarize those benchmarks relative to validation needs in shielding applications currently found within the ICSBEP Handbook. The current and future plans of shielding benchmarks with regards to the ICSBEP will also be discussed.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2025

Impacts of benchmarking choices on inferred model skill of the Arctic–Boreal terrestrial carbon cycle

Abstract Land surface models require continuous validation against observations to improve and reduce simulation uncertainty. However, inferred model performance can be heavily influenced by subjective choices made in the selection and application of observational data products. A key area often misrepresented by models is the Arctic–Boreal region, which is a potential tipping point region in Earth’s climate system due to large permafrost carbon stocks that are vulnerable to release with climate warming. We use the International Land Model Benchmarking (ILAMB) framework to evaluate how the model skill of TRENDY-v9 models varies based on the choice of observational-based benchmark and how benchmarks are applied in model evaluation. This analysis uses global datasets integrated into ILAMB and new, regionally-specific observational products from the Arctic–Boreal Vulnerability Experiment. Our results cover the overall time period of 1979–2019 and show that model scores can vary substantially depending on the data product applied, with higher model scores indicating better model performance against observations. The lowest model scores occur when benchmarked against regional, compared to global, datasets. We also evaluate observed and modeled functional relationships between ecosystem respiration and air temperature and between gross primary production and precipitation. Here, we find that the magnitude and shape of the responses are strongly impacted by the choice of observational dataset and the approach used to construct the functional relationship benchmark. These results suggest that model evaluation studies could conclude a false sense of model skill if only using a single benchmark data product or if not applying regional data products when performing a regional model analysis. Collectively, our findings highlight the influence of benchmarking choices on model evaluation and point to the need for benchmarking guidelines when assessing model skill.

Poe, Jeralyn (ORCID:0000000318495278)↗
DOE OSTI2020

Relevant Advanced Reactor Benchmarks for Nuclear Data Assessment

Advanced reactor concepts currently being developed throughout the industry are significantly different from light water reactor (LWR) designs with respect to geometry, materials, and operating conditions, and consequently, with respect to their reactor physics behavior. Given the limited operating experience with non-LWRs, the accurate simulation of reactor physics and the quantification of associated uncertainties are critical for ensuring that advanced reactor concepts operate within the appropriate safety margins. Nuclear data are a major source of input uncertainties in reactor physics analysis. As part of an ongoing project at Oak Ridge National Laboratory, the effects of nuclear data uncertainties on key figures of merit associated with advanced reactor safety are being assessed for selected advanced reactor technologies. Key nuclear data relevant for reactor safety analysis for each selected advanced reactor technology were identified in Phase 1, and their impact on important key figures of merit was assessed in Phase 2. This report describes the outcome of Phase 3. Available benchmarks and fuel irradiation data for use in evaluating the impact of uncertainties and gaps in nuclear data that impact reactivity control for advanced reactor designs through the fuel cycle were identified and assessed. Benchmarks were identified by searching (1) the Organisation for Economic Co-operation and Development (OECD)/Nuclear Energy Agency (NEA) International Criticality Safety Benchmark Evaluation Project (IRPhEP) handbook, (2) the OCED/NEA International Reactor Physics Experiment Evaluation Project (IRPhEP) handbook, (3) ongoing OECD/NEA benchmark activities, and (4) documentation in public literature. Relevant benchmarks were identified by selecting reactors with geometry, materials, and neutron energy spectra similar to those of selected advanced reactor technologies. This assessment identified six benchmarks, of which three are experimental and three are purely computational. One experimental and one computation benchmark contain depleted fuel; all others are limited to fresh fuel. This report provides short descriptions of the selected benchmarks along with the availability of measured data for comparison.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2022

IER 555: Godiva Benchmark Update CED-2 (Final Design Report)

The International Criticality Safety Benchmark Evaluation Project (ICSBEP) evaluation of the Godiva IV critical assembly, HEU-MET-FAST-086: GODIVA-IV DELAYED-CRITICAL EXPERIMENTS (HMF-086), was completed by Russ Mosteller. Five critical experiment configurations performed at the Los Alamos National Laboratory (LANL) Technical Area (TA)-18 were evaluated as acceptable benchmark cases. The five cases consist of four delayed critical configurations which differ in control rod positions and one prompt critical configuration. All cases calculated a lower $k_{eff}$ than measured by experiment. This data is referred to as the TA-18 Godiva IV benchmark in this report. In 2005, Godiva IV was disassembled for relocation to the Nevada Test Site (NTS), now Nevada National Security Site (NNSS), at the National Criticality Experiments Research Center (NCERC). Following the disassembly and subsequent reassembly and startup of Godiva IV at NCERC, additional information about the Godiva IV components was obtained. An errata note was added to the HMF-086 evaluation in the ICSBEP handbook to provide this new information until a revision to the benchmark evaluation could be performed. In addition to those corrections, there are differences between the Godiva IV assembly at TA-18 and the Godiva IV assembly at NCERC. These differences include both assembly-specific differences (differences in the safety block gap, differences in the control rod positions, a new NCERC Top Hat and contamination shield) as well as environmental differences, such as the size and shape of the experimental building where the assembly is located. An additional model with similar cases, referred to as the NCERC Godiva IV benchmark in this report, will be added to the revised HMF-086 to capture these additional differences. This will provide the best benchmark model of Godiva for use by those performing experiments at NCERC. The IER 555 CED-2 report documents the information that will be updated in the HMF-086 revision, both the corrections to the TA-18 Godiva IV benchmark and the subsequent changes to create a NCERC Godiva IV benchmark. It describes the measurements that will be performed for cases similar to those performed at TA-18. It also describes measurements that will be included in the evaluation as additional data.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2020

Generation IV Benchmarking of TRISO Fuel Performance Models Under Accident Conditions Final Report

The Generation IV International Forum (GIF) is a co-operative international endeavor of fourteen members organized to carry out the research and development needed to establish the feasibility and performance capabilities of the next generation nuclear energy systems. GIF selected six reactor technologies, amongst which is the Very High Temperature Reactor (VHTR) that is primarily dedicated to the cogeneration of electricity and hydrogen. The technical basis for VHTR is the tristructural isotropic (TRISO)-coated particle fuel, the graphite as the core structure, helium coolant, as well as the dedicated core layout and lower power density to removal decay heat in a natural way. At the heart of safety features of the VHTR concept lie the TRISO fuel particles that are designed to keep their structural integrity and retain fission products at temperatures up to 1600°C. As part as the design and future operation of VHTRs, a key aspect is the accurate prediction of fuel performance under irradiation and accident conditions. Modeling and simulation allow prediction of TRISO fuel behavior when subject to neutron flux and in high temperature accident scenarios. The refinement of the fuel performance models and codes is performed by comparison to in-pile and out-of-pile experimental data that reproduce the expected irradiation conditions in high temperature gas-cooled reactors (HTGRs). Historically, the International Atomic Energy Agency (IAEA) developed a benchmark dedicated to the validation of predictive methods for fuel and fission product behavior through the Coordinated Research Program CRP-2 (IAEA, 1997). CRP-2 was later updated to cover fuel fabrication, quality assurance, irradiation performance, safety testing, and spent fuel. The scope of the resulting CRP-6 benchmarks focused on HTGR fuel performance and fission product release (IAEA, 2012). Taking advantage of additional TRISO fuel fabrication, irradiation, and safety testing campaigns, GIF launched a Generation IV Benchmarking of TRISO Fuel Performance Models under Accident Conditions in late 2015. This GIF benchmark is a three-year program steered by Idaho National Laboratory (INL, USA). The other participants include the Japan Atomic Energy Agency (JAEA, Japan) and the Korea Atomic Energy Research Institute (KAERI, Korea). The objectives of the benchmark are to: follow on the IAEA CRP benchmarks, (2) model fission product release under accident conditions, (3) compare results obtained by the fuel performance modeling codes of the benchmark participants, and (4) compare these code predictions to experimental data. Safety tests chosen for modeling include the first and second experiments of the Advanced Gas Reactor program (AGR-1 and AGR-2) and the High Flux Reactor (HFR) EU1bis experiment. This report presents the results obtained by the three research institutions using their respective fuel performance modeling codes. Comparisons of the corresponding fission product release predictions are made with experimental data from AGR-1, AGR-2, and HFR-EU1bis. The benchmark results show good agreements between all participants but also show a general trend of over-prediction of the experimental release data, which is mainly attributed to the use of over-estimated diffusion coefficients.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2025

Results of a Geant4 benchmarking study for bio‐medical applications, performed with the G4‐Med system

Geant4, a Monte Carlo Simulation Toolkit extensively used in bio-medical physics, is in continuous evolution to include newest research findings to improve its accuracy and to respond to the evolving needs of a very diverse user community. In 2014, the G4-Med benchmarking system was born from the effort of the Geant4 Medical Simulation Benchmarking Group, to benchmark and monitor the evolution of Geant4 for medical physics applications. The G4-Med system was first described in our Medical Physics Special Report published in 2021. Results of the tests were reported for Geant4 10.5. Purpose In this work, we describe the evolution of the G4-Med benchmarking system. Methods The G4-Med benchmarking suite currently includes 23 tests, which benchmark Geant4 from the calculation of basic physical quantities to the simulation of more clinically relevant set-ups. New tests concern the benchmarking of Geant4-DNA physics and chemistry components for regression testing purposes, dosimetry for brachytherapy with a 125 I source, dosimetry for external x-ray and electron FLASH radiotherapy, experimental microdosimetry for proton therapy, and in vivo PET for carbon and oxygen beams. Regression testing has been performed between Geant4 10.5 and 11.1. Finally, a simple Geant4 simulation has been developed and used to compare Geant4 EM physics constructors and physics lists in terms of execution times. Results In summary, our EM tests show that the parameters of the multiple scattering in the Geant4 EM constructor G4EmStandardPhysics_option3 in Geant4 11.1, while improving the modeling of the electron backscattering in high atomic number targets, are not adequate for dosimetry for clinical x-ray and electron beams. Therefore, these parameters have been reverted back to those of Geant4 10.5 in Geant4 11.2.1. The x-ray radiotherapy test shows significant differences in the modeling of the bremsstrahlung process, especially between G4EmPenelopePhysics and the other constructors under study (G4EmLivermorePhysics, G4EmStandardPhysics_option3, and G4EmStandardPhysics_option4). These differences will be studied in an in-depth investigation within our Group. Improvement in Geant4 11.1 has been observed for the modeling of the proton and carbon ion Bragg peak with energies of clinical interest, thanks to the adoption of ICRU90 to calculate the low energy proton stopping powers in water and of the Linhard–Sorensen ion model, available in Geant4 since version 11.0. Nuclear fragmentation tests of interest for carbon ion therapy show differences between Geant4 10.5 and 11.1 in terms of fragment yields. In particular, a higher production of boron fragments is observed with Geant4 11.1, leading to a better agreement with reference data for this fragment. Conclusions Based on the overall results of our tests, we recommend to use G4EmStandardPhysics_option4 as EM constructor and QGSP_BIC_HP with G4EmStandardPhysics_option4, for hadrontherapy applications. The Geant4-DNA physics lists report differences in modeling electron interactions in water, however, the tests have a pure regression testing purpose so no recommendation can be formulated.

62 RADIOLOGY AND NUCLEAR MEDICINE↗
DOE OSTI2026

A New Shutdown Dose Rate Benchmark Problem for Representative Fusion Applications

Here, this work introduces a new benchmark problem for calculating shutdown dose rates (SDDRs) aimed at fusion reactor applications. The model is designed to represent a simplified version of a typical ITER port plug. The responses of interest include neutron flux, gamma flux, and gamma SDDR at 12 different locations scattered throughout the port. This article outlines the geometry specifications of the problem, provides material definitions for the components, specifies the required responses to be calculated, and presents the source definition information. The need for this benchmark arises from the limited availability of publicly accessible references, with only one benchmark representing the typical dimensions and materials found in fusion systems. This existing benchmark has been cited extensively, reflecting the demand within the scientific community to test both established and novel workflows for SDDR calculations. However, since its presentation at a conference in 2011, the results have become increasingly well known. Moreover, the absence of formal publication and peer review has led to the details of this benchmark being extracted from secondary sources, such as subsequent studies that reference it. As a result, analysts are left with significant flexibility in interpreting the key parameters, which can be adjusted to account for unknown systematic errors, ultimately reproducing the already well-known responses. This new benchmark serves as an updated version of that earlier work, with the aim of providing a more reliable description of the materials and their impurities, which is crucial for assessing activation and subsequent gamma emission. Additionally, it seeks to provide a geometry that more closely represents an ITER port plug. The improvements in the problem definition will lead to a more reproducible benchmark problem, while also presenting the radiation transport community with a completely new challenge. The results will be published in a future article to allow analysts adequate time to analyze this problem independently.

Benchmark↗
DOE OSTI2025

An HPC benchmark survey and taxonomy for characterization

The field of High-Performance Computing (HPC) is defined by providing computing devices with highest performance for a variety of demanding scientific users. The tight co-design relationship between HPC providers and users propels the field forward, paired with technological improvements, achieving continuously higher performance and resource utilization. A key device for system architects, architecture researchers, and scientific users are benchmarks, allowing for well-defined assessment of hardware, software, and algorithms. Many benchmarks exist in the community, from individual niche benchmarks testing specific features, to large-scale benchmark suites for whole procurements. We survey the available HPC benchmarks, summarizing them in table form with key details and concise categorization, also through an interactive website. For categorization, we present a benchmark taxonomy for well-defined characterization of benchmarks.

Benchmarking↗
DOE OSTI2025

Verification of the Uniformly-Ordered Binary Decision Algorithm in Correlated-Benchmark Whisper Calculations

Whisper is a nuclear criticality safety code package that aids analysts in validation exercises by computing upper subcritical limits (USL) for applications of interest. To obtain statistically meaningful, significant, and conservative USLs, the analyst must ensure that Whisper selects a sufficient number of benchmarks that are neutronically similar to the application. Many of the available benchmarks are correlated but are currently treated as independent, leading to an artificially small sample size, as their individual information contributions will be overestimated. To aid the analyst in obtaining a sufficient sample size, prior work [2] demonstrated application of the Uniformly-Ordered Binary Decision (UOBD) algorithm in adjusting benchmark weights to account for benchmark correlations. This work provides verification of the Whisper implementation and considers the impact of updated benchmark correlations compared to those available previously. We demonstrate that the UOBD algorithm performs as expected with an analytic example. With HEU-SOL-THERM-001 cases 1 through 10 as the applications, we compare the USLs computed with benchmark correlations available in the Whisper 1.1 release only to those computed with additional benchmark correlations from DICE 2023 and demonstrate substantive differences.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2026

Systematic Benchmarking of Climate Models: Methodologies, Applications, and New Directions

As climate models become increasingly complex, there is a growing need to comprehensively and systematically assess model performance with respect to observations. Given the increasing number and diversity of climate model simulations in use, the community has moved beyond simple model intercomparison and toward developing methods capable of benchmarking a large number of simulations against a suite of climate metrics. Here, we present a detailed review of evaluation and benchmarking methods and approaches developed in the last decade, focusing primarily on scientific implications for Coupled Model Intercomparison Project (CMIP) simulations and CMIP6 results that contributed to the Intergovernmental Panel on Climate Change (IPCC) Sixth Assessment Report (AR6). Based on this review, we explain the resulting contemporary philosophy of model benchmarking, and provide clear distinctions and definitions of the terms model verification, process validation, evaluation, and benchmarking. While significant progress has been made in model development based on systematic evaluation and benchmarking efforts, some climate system biases still remain. The development of open‐source community software packages has played a fundamental role in identifying areas of significant model improvement and bias reduction. We review the key features of several software packages that have been commonly used over the past decade to evaluate and benchmark global and regional climate models. Additionally, we discuss best practices for the selection of evaluation and benchmarking metrics and for interpreting the obtained results, the importance of selecting suitable sources of reference data and accurate uncertainty quantification.

Environmental sciences↗
DOE OSTI2024

JARVIS-Leaderboard: a large scale benchmark of materials design methods

Abstract Lack of rigorous reproducibility and validation are significant hurdles for scientific development across many fields. Materials science, in particular, encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC), and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: https://pages.nist.gov/jarvis_leaderboard/

36 MATERIALS SCIENCE↗
DOE OSTI2023

Outcomes of WPEC SG47 on "Use of Shielding Integral Benchmark Archive and Database for Nuclear Data Validation"

The Working Party on International Nuclear Data Evaluation Co-operation Subgroup 47 (WPECSG47) entitled "Use of Shielding Integral Benchmark Archive and Database for Nuclear Data Validation" was organised from 2019 and 2022 with the objectives to promote more systematic and wider use of shielding benchmark experiments in nuclear data (ND) and transport code validation and development, to provide feedback on the Shielding Integral Benchmark Archive and Database (SINBAD), and to promote its further development in coordination with the Expert Group on Physics of Reactor Systems (EGPRS). Altogether 9 meetings, the large majority (8) held remotely, were organised during the past 3 years to discuss the experience on the use of SINBAD, evaluation of new benchmarks and improvements to be contributed to the database which was severely neglected and lacking maintenance over the past ← 10+ years. Several proposals for new or updated benchmark evaluation were presented and discussed, such as FNG copper, LLNL pulsed spheres, CIAE iron sphere, KFK 1977 gamma measurements, Rez Fe sphere, ASPIS, ORNL Oxygen broomstick, TIARA and others. Complementing the database with new features was also discussed, for example providing the nuclear data sensitivity profiles more systematically would facilitate and better guide the use of data. Information on the geometry, (radiation source) and materials available in CAD format is expected to allow an easier and less error prone reference for computational model preparation and a potential input to CAD based workflows. Inputs for various transport codes and other benchmark data from participants have been shared via the NEA GitLab which could hopefully in the future evolve and form a bases for critically checked and validated benchmark data. Future development of SINBAD will be monitored by EGPRS and the newly created SINBAD Task Force.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Ecological Benchmark for Chemicals

The Ecological Benchmark Tool dataset serves as a comprehensive repository of benchmarks designed to assess ecological risks at contaminated sites. This tool facilitates the evaluation of various environmental media and contaminants, supporting regulatory compliance and ecological protection. Benchmarks are available for air, biota, sediment, soil, and surface water. The dataset also provides species-specific benchmarks for fish, plants, birds, mammals, and invertebrates. Users can select benchmark sources, media, individual chemicals, and retrieve results in tabular or spreadsheet formats for analysis. Benchmarks are derived from authoritative sources, including government agencies, scientific councils, and academic publications. The dataset supports ecological risk assessments, regulatory decision-making, and environmental planning, with tools for benchmarking against chemical thresholds, sensitive species protection, and habitat impact evaluations. This structured approach ensures a robust evaluation of ecological risks tailored to site-specific and regulatory needs.

Stewart, Debra [Oak Ridge National Laboratory (ORN↗
DOE OSTI2024

Ecological Benchmark for Radionuclides

The Ecological Benchmark Tool for radionuclides dataset serves as a comprehensive repository of benchmarks designed to assess ecological risks at contaminated sites. This tool facilitates the evaluation of various environmental media and contaminants, supporting regulatory compliance and ecological protection. Benchmarks are available for sediment, soil, and surface water. The dataset also provides species-specific benchmarks for fish, plants, birds, mammals, and invertebrates. Users can select benchmark sources, media, individual radionuclides, and retrieve results in tabular or spreadsheet formats for analysis. Benchmarks are derived from authoritative sources, including government agencies, scientific councils, and academic publications. The dataset supports ecological risk assessments, regulatory decision-making, and environmental planning, with tools for benchmarking against radiological thresholds, sensitive species protection, and habitat impact evaluations. This structured approach ensures a robust evaluation of ecological risks tailored to site-specific and regulatory needs.

Stewart, Debra [Oak Ridge National Laboratory (ORN↗