Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Benchmarking”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

Systematic Benchmarking of Climate Models: Methodologies, Applications, and New Directions

As climate models become increasingly complex, there is a growing need to comprehensively and systematically assess model performance with respect to observations. Given the increasing number and diversity of climate model simulations in use, the community has moved beyond simple model intercomparison and toward developing methods capable of benchmarking a large number of simulations against a suite of climate metrics. Here, we present a detailed review of evaluation and benchmarking methods and approaches developed in the last decade, focusing primarily on scientific implications for Coupled Model Intercomparison Project (CMIP) simulations and CMIP6 results that contributed to the Intergovernmental Panel on Climate Change (IPCC) Sixth Assessment Report (AR6). Based on this review, we explain the resulting contemporary philosophy of model benchmarking, and provide clear distinctions and definitions of the terms model verification, process validation, evaluation, and benchmarking. While significant progress has been made in model development based on systematic evaluation and benchmarking efforts, some climate system biases still remain. The development of open‐source community software packages has played a fundamental role in identifying areas of significant model improvement and bias reduction. We review the key features of several software packages that have been commonly used over the past decade to evaluate and benchmark global and regional climate models. Additionally, we discuss best practices for the selection of evaluation and benchmarking metrics and for interpreting the obtained results, the importance of selecting suitable sources of reference data and accurate uncertainty quantification.

Environmental sciences↗

JARVIS-Leaderboard: a large scale benchmark of materials design methods

Abstract Lack of rigorous reproducibility and validation are significant hurdles for scientific development across many fields. Materials science, in particular, encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC), and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: https://pages.nist.gov/jarvis_leaderboard/

36 MATERIALS SCIENCE↗

Outcomes of WPEC SG47 on "Use of Shielding Integral Benchmark Archive and Database for Nuclear Data Validation"

The Working Party on International Nuclear Data Evaluation Co-operation Subgroup 47 (WPECSG47) entitled "Use of Shielding Integral Benchmark Archive and Database for Nuclear Data Validation" was organised from 2019 and 2022 with the objectives to promote more systematic and wider use of shielding benchmark experiments in nuclear data (ND) and transport code validation and development, to provide feedback on the Shielding Integral Benchmark Archive and Database (SINBAD), and to promote its further development in coordination with the Expert Group on Physics of Reactor Systems (EGPRS). Altogether 9 meetings, the large majority (8) held remotely, were organised during the past 3 years to discuss the experience on the use of SINBAD, evaluation of new benchmarks and improvements to be contributed to the database which was severely neglected and lacking maintenance over the past ← 10+ years. Several proposals for new or updated benchmark evaluation were presented and discussed, such as FNG copper, LLNL pulsed spheres, CIAE iron sphere, KFK 1977 gamma measurements, Rez Fe sphere, ASPIS, ORNL Oxygen broomstick, TIARA and others. Complementing the database with new features was also discussed, for example providing the nuclear data sensitivity profiles more systematically would facilitate and better guide the use of data. Information on the geometry, (radiation source) and materials available in CAD format is expected to allow an easier and less error prone reference for computational model preparation and a potential input to CAD based workflows. Inputs for various transport codes and other benchmark data from participants have been shared via the NEA GitLab which could hopefully in the future evolve and form a bases for critically checked and validated benchmark data. Future development of SINBAD will be monitored by EGPRS and the newly created SINBAD Task Force.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Combining Phase Identification and Statistic Modeling for Automated Parallel Benchmark Generation

Parallel application benchmarks are indispensable for evaluating/optimizing HPC software and hardware. However, it is very challenging and costly to obtain high-fidelity benchmarks reflecting the scale and complexity of state-of-the-art parallel applications. Hand-extracted synthetic benchmarks are time-and labor-intensive to create. Real applications themselves, while offering most accurate performance evaluation, are expensive to compile, port, reconfigure, and often plainly inaccessible due to security or ownership concerns. This work contributes APPRIME, a novel tool for trace-based automatic parallel benchmark generation. Taking as input standard communication-I/O traces of an application's execution, it couples accurate automatic phase identification with statistical regeneration of event parameters to create compact, portable, and to some degree reconfigurable parallel application benchmarks. Experiments with four NAS Parallel Benchmarks (NPB) and three real scientific simulation codes confirm the fidelity of APPRIME benchmarks. They retain the original applications' performance characteristics, in particular the relative performance across platforms.

Jin, Ye↗

Ecological Benchmark for Chemicals

The Ecological Benchmark Tool dataset serves as a comprehensive repository of benchmarks designed to assess ecological risks at contaminated sites. This tool facilitates the evaluation of various environmental media and contaminants, supporting regulatory compliance and ecological protection. Benchmarks are available for air, biota, sediment, soil, and surface water. The dataset also provides species-specific benchmarks for fish, plants, birds, mammals, and invertebrates. Users can select benchmark sources, media, individual chemicals, and retrieve results in tabular or spreadsheet formats for analysis. Benchmarks are derived from authoritative sources, including government agencies, scientific councils, and academic publications. The dataset supports ecological risk assessments, regulatory decision-making, and environmental planning, with tools for benchmarking against chemical thresholds, sensitive species protection, and habitat impact evaluations. This structured approach ensures a robust evaluation of ecological risks tailored to site-specific and regulatory needs.

Stewart, Debra [Oak Ridge National Laboratory (ORN↗

Ecological Benchmark for Radionuclides

The Ecological Benchmark Tool for radionuclides dataset serves as a comprehensive repository of benchmarks designed to assess ecological risks at contaminated sites. This tool facilitates the evaluation of various environmental media and contaminants, supporting regulatory compliance and ecological protection. Benchmarks are available for sediment, soil, and surface water. The dataset also provides species-specific benchmarks for fish, plants, birds, mammals, and invertebrates. Users can select benchmark sources, media, individual radionuclides, and retrieve results in tabular or spreadsheet formats for analysis. Benchmarks are derived from authoritative sources, including government agencies, scientific councils, and academic publications. The dataset supports ecological risk assessments, regulatory decision-making, and environmental planning, with tools for benchmarking against radiological thresholds, sensitive species protection, and habitat impact evaluations. This structured approach ensures a robust evaluation of ecological risks tailored to site-specific and regulatory needs.

Stewart, Debra [Oak Ridge National Laboratory (ORN↗

Evaluation of CHO Benchmarks on the Arria 10 FPGA using Intel FPGA SDK for OpenCL

The OpenCL standard is an open programming model for accelerating algorithms on heterogeneous computing system. OpenCL extends the C-based programming language for developing portable codes on different platforms such as CPU, Graphics processing units (GPUs), Digital Signal Processors (DSPs) and Field Programmable Gate Arrays (FPGAs). The Intel FPGA SDK for OpenCL is a suite of tools that allows developers to abstract away the complex FPGA-based development flow for a high-level software development flow. Users can focus on the design of hardware-accelerated kernel functions in OpenCL and then direct the tools to generate the low-level FPGA implementations. The approach makes the FPGA-based development more accessible to software users as the needs for hybrid computing using CPUs and FPGAs are increasing. It can also significantly reduce the hardware development time as users can evaluate different ideas with high-level language without deep FPGA domain knowledge. Benchmarking of OpenCL-based framework is an effective way for analyzing the performance of system by studying the execution of the benchmark applications. CHO is a suite of benchmark applications that provides support for OpenCL [1]. The authors presented CHO as an OpenCL port of the CHStone benchmark. Using Altera OpenCL (AOCL) compiler to synthesize the benchmark applications, they listed the resource usage and performance of each kernel that can be successfully synthesized by the compiler. In this report, we evaluate the resource usage and performance of the CHO benchmark applications using the Intel FPGA SDK for OpenCL and Nallatech 385A FPGA board that features an Arria 10 FPGA device. The focus of the report is to have a better understanding of the resource usage and performance of the kernel implementations using Arria-10 FPGA devices compared to Stratix-5 FPGA devices. In addition, we also gain knowledge about the limitations of the current compiler when it fails to synthesize a benchmark application.

97 MATHEMATICS AND COMPUTING↗

GEN-IV Benchmarking of Triso Fuel Performance Models under accident conditions modeling input data

This document presents the benchmark plan for the calculation of particle fuel performance on safety testing experiments that are representative of operational accidental transients. The benchmark is dedicated to the modeling of fission product release under accident conditions by fuel performance codes from around the world, and the subsequent comparison to post-irradiation experiment (PIE) data from the modeled heating tests. The accident condition benchmark is divided into three parts: • The modeling of a simplified benchmark problem to assess potential numerical calculation issues at low fission product release. • The modeling of the AGR-1 and HFR-EU1bis safety testing experiments. • The comparison of the AGR-1 and HFR-EU1bis modeling results with PIE data. The simplified benchmark case, thereafter named NCC (Numerical Calculation Case), is derived from “Case 5” of the International Atomic Energy Agency (IAEA) Coordinated Research Program (CRP) on coated particle fuel technology [IAEA 2012]. It is included so participants can evaluate their codes at low fission product release. “Case 5” of the IAEA CRP-6 showed large code-to-code discrepancies in the release of fission products, which were attributed to “effects of the numerical calculation method rather than the physical model” [IAEA 2012]. The NCC is therefore intended to check if these numerical effects subsist. The first two steps imply the involvement of the benchmark participants with a modeling effort following the guidelines and recommendations provided by this document. The third step involves the collection of the modeling results by Idaho National Laboratory (INL) and the comparison of these results with the available PIE data. The objective of this document is to provide all necessary input data to model the benchmark cases, and to give some methodology guidelines and recommendations in order to make all results suitable for comparison with each other. The participants should read this document thoroughly to make sure all the data needed for their calculations is provided in the document. Missing data will be added to a revision of the document if necessary. 09/2016: Tables 6 and 8 updated. AGR-2 input data added

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Computational and Experimental Benchmarking for Transient Fuel Testing

In 2018 the Transient Reactor Test (TREAT) Facility was restarted. This now-operational facility brings back world-class transient fuel testing capabilities to the U.S. Among a number of parallel efforts prior to the TREAT Facility’s restart a Department of Energy (DoE) Integrated Research Project (IRP) was funded for the purpose of benchmarking the TREAT with several goals in mind: To produce a comprehensive evaluation of existing TREAT Facility neutronics data using the next generation reactor core neutronics codes in accordance with established guidelines per the International Handbook of Evaluated Reactor Physics Benchmark Experiments (IRPhEP). To perform a benchmark using ready-use computational fluid dynamics tools against that of a historically collected sodium loop experiment collected in the TREAT. To design, develop and test a new experimental thermal hydraulic system of a representative TREAT Facility water flow loop. And benchmark the data collected against that produced via RELAP5-3D and TRACE. To develop a comprehensive instrumentation plan for the TREAT Facility by benchmarking performing of advanced in-pile instruments in the Massachusetts Institute of Technology Research Reactor (MITR) and the TREAT. The integrated team brought together collaborating members from three universities including Oregon State University (OSU), the University of Michigan (UM), and Massachusetts Institute of Technology (MIT); three national laboratories, Idaho National Laboratory, Argonne National Laboratory (ANL), and Oak Ridge National Laboratory (ORNL); and two industry partners, TerraPower (TP) and Harris Thermal Transfer Products (HTTP). A brief summary of accomplishments resulting from this IRP includes two filed patents, more than 40 conference papers, two published benchmark reports through IRPhEP, a new representative thermal hydraulic test facility alike the in-pile PWR water flow loop for TREAT and the first advanced instruments tested within the TREAT core upon its restart. The papers presented herein were developed and provide a summary of the significant technical accomplishments which align with the four project’s objectives. The collaborators has significant support from countless parties including the operational and technical staff at TREAT to ensure that their outcomes most impactfully aligned with program needs. Lastly the collaborators would like to acknowledge the fiscal support of the DoE’s Office of Nuclear Energy under work package identification number NU-15-OR-OSU- 0701-01. This report contains selected papers from the Computational and Experimental Benchmarking for Transient Fuel Testing Integrated Research Project.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Molten Salt Reactor Experiment Benchmark Evaluation

As one of the six advanced reactor concepts selected by the 2002 Generation IV roadmap as a technology meriting future research, Molten Salt Reactors (MSRs) attracted broad attention and multiple private and public entities are working towards its commercialization. Facing stringent regulatory requirements, validations of computational codes used to calculate and prove the safety characteristics of MSRs play a key role. This project developed the world-first, MSR-related reactor physics benchmark based on a series of zero-power experiments conducted at the Molten Salt Reactor Experiment (MSRE). The benchmark was successfully reviewed by the International Reactor Physics Experiment Evaluation Project (IRPhEP) committee and is available in the IRPhEP handbook starting form the 2019 edition. To benchmark experimental data from the MSRE, a three-dimensional high-fidelity model was developed using the Monte Carlo neutron transport code Serpent 2 and ad-hoc methods have been implemented to account for the unique feature of fuel salt motion. The calculated effective multiplication factor, k eff , for the first criticality experiment, when 235 U was progressively added to the fuel salt in order to achieve criticality with stationary salt and isothermal conditions, was 1.02132 (±3 pcm). The total uncertainty for experimental k eff was estimated to be 420 pcm. The calculated k eff is 2.154% larger than the experimental and benchmark model value, which is approximately five times the benchmark model uncertainty. It is to be noted that, for systems containing large volume of graphite (or other carbonaceous materials), Monte Carlo codes tend to overestimate the keff of the benchmark model by 1% to 2%. The bias is possibly due to uncertainties in the impurity content of the graphite blocks, uncertainty in the neutron capture cross section of carbon or uncertainty in the properties of nuclear-grade graphite. The calculated reactivity coefficient of 235 U concentration (0.2228 ± 0.0014, represented as the change of reactivity over the relative change of 235 U mass in loop) matches well with the experimental value (0.223 ± 0.006), strengthening the confidence on the accurate representation of the fuel salt composition in the MSRE benchmark. Additional reactivity effect calculations, including the control rod bank worth, reactivity effects of fuel circulation and isothermal and fuel temperature coefficients show good agreement with the experimental values (within 1σ) as well.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Q1 2023 U.S. Solar Photovoltaic System and Energy Storage Cost Benchmarks With Minimum Sustainable Price Analysis Data File

The U.S. Department of Energy's (DOE's) Solar Energy Technologies Office (SETO) aims to accelerate the advancement and deployment of solar technology in support of an equitable transition to a decarbonized economy no later than 2050, starting with a decarbonized power sector by 2035. Its approach to achieving this goal includes driving innovations in technology, hardware, and soft cost reductions to make solar affordable and accessible for all. As part of this effort, SETO must track solar cost trends so it can focus its research and development (R&D) on the highest-impact activities. The benchmarks in this report are bottom-up cost estimates of all major inputs to PV and energy storage system installations. Bottom-up costs are based on national averages and do not necessarily represent typical costs in all local markets. Like last year's report, this year's report includes two distinct sets of benchmarks: minimum sustainable price (MSP) benchmarks and modeled market price (MMP) benchmarks. MSP benchmarks can be interpreted as the minimum price a company needs to charge to remain financially solvent in the long term based on the minimum sustainable prices of all inputs including minimum sustainable profit margins. MMP benchmarks can be interpreted as the actual cash sales price a company charges in the given benchmark period. These simplified estimates are useful for tracking technological progress, but they do not reflect all experiences. In fact, no individual estimate under any approach can reflect the diversity of the PV and storage manufacturing and installation industries.

14 SOLAR ENERGY↗

Preliminary Benchmark Evaluation of Japan’s High Temperature Engineering Test Reactor

A benchmark model of the initial fully-loaded start-up core critical of Japan’s High Temperature Engineering Test Reactor (HTTR) was developed to provide data in support of ongoing validation efforts of the Very High Temperature Reactor Program using publicly available resources. The HTTR is a 30 MWt test reactor utilizing graphite moderation, helium coolant, and prismatic TRISO fuel. The benchmark was modeled using MCNP5 with various neutron cross-section libraries. An uncertainty evaluation was performed by perturbing the benchmark model and comparing the resultant eigenvalues. The calculated eigenvalues are approximately 2-3% greater than expected with an uncertainty of ±0.70%. The primary sources of uncertainty are the impurities in the core and reflector graphite. The release of additional HTTR data could effectively reduce the benchmark model uncertainties and bias. Sensitivity of the results to the graphite impurity content might imply that further evaluation of the graphite content could significantly improve calculated results. Proper characterization of graphite for future Next Generation Nuclear Power reactor designs will improve computational modeling capabilities. Current benchmarking activities include evaluation of the annular HTTR cores and assessment of the remaining start-up core physics experiments, including reactivity effects, reactivity coefficient, and reaction-rate distribution measurements. Long term benchmarking goals might include analyses of the hot zero-power critical, rise-to-power tests, and other irradiation, safety, and technical evaluations performed with the HTTR.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Preliminary Assessment of ATR-C Capabilities to Provide Integral Benchmark Data for Key Structural/Matrix Materials that May be Used for Nuclear Data Testing and Analytical Methods Validation

The purpose of this document is to identify some suggested types of experiments that can be performed in the Advanced Test Reactor Critical (ATR-C) facility. A fundamental computational investigation is provided to demonstrate possible integration of experimental activities in the ATR-C with the development of benchmark experiments. Criticality benchmarks performed in the ATR-C could provide integral data for key matrix and structural materials used in nuclear systems. Results would then be utilized in the improvement of nuclear data libraries and as a means for analytical methods validation. It is proposed that experiments consisting of well-characterized quantities of materials be placed in the Northwest flux trap position of the ATR-C. The reactivity worth of the material could be determined and computationally analyzed through comprehensive benchmark activities including uncertainty analyses. Experiments were modeled in the available benchmark model of the ATR using MCNP5 with the ENDF/B-VII.0 cross section library. A single bar (9.5 cm long, 0.5 cm wide, and 121.92 cm high) of each material could provide sufficient reactivity difference in the core geometry for computational modeling and analysis. However, to provide increased opportunity for the validation of computational models, additional bars of material placed in the flux trap would increase the effective reactivity up to a limit of 1$ insertion. For simplicity in assembly manufacture, approximately four bars of material could provide a means for additional experimental benchmark configurations, except in the case of strong neutron absorbers and many materials providing positive reactivity. Future tasks include the cost analysis and development of the experimental assemblies, including means for the characterization of the neutron flux and spectral indices. Oscillation techniques may also serve to provide additional means for experimentation and validation of computational methods and acquisition of integral data for improving neutron cross sections. Further assessment of oscillation techniques for implementation in the ATR-C may be of additional benefit. The establishment of benchmark experiment capabilities in the ATR-C would allow for the further development of techniques and facility enhancements involving neutronics experimentation in the ATR-C.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Beyond Energy Efficiency: A clustering approach to embed demand flexibility into building energy benchmarking

The intermittency of carbon-free renewables and the demand changes associated with the widespread push for electrifying the transportation and building sectors provides an opportunity for buildings to go beyond energy efficiency and push towards providing demand flexibility to the electricity grid. The duality of energy efficiency and demand flexibility is necessary for success in a sustainable and reliable energy transition. Current building energy benchmarking models are limited in their ability to integrate concepts of demand flexibility and/or utilize granular smart meter data. Thus, current benchmarking methods are focused annual energy usage and fail to incorporate how the time of use of energy consumption impacts emissions in a quickly changing energy grid. Without a more comprehensive view of energy usage and associated real-time emissions, current benchmarking methods are unlikely to realize the full decarbonization potential of buildings. New emerging data streams provide an opportunity to develop a new generation of benchmarking energy models that embed dimensions of energy efficiency, grid interactivity, and demand flexibility into their analysis. In this paper, we propose a four-step method for embedding grid interactivity and demand flexibility into building benchmarking models that utilizes emerging building and time-series electricity data streams. We first engineer features to produce a mix-type dataset that encompasses many attributes of grid-interactive and efficient buildings, and then we apply K-medoids using Gower's Distance to produce peer-group clusters. We apply the method to a case study of 306 primary and secondary schools in southern California, USA. The results show that the method effectively clusters buildings by attributes of demand flexibility and energy efficiency. The clustering results reveal patterns in inefficient building operations and demand inflexibility at the building peer group level. In conclusion, the interpretation of clusters can serve as an integrated energy efficiency and demand flexibility benchmarking model and inform performance-specific policy targeting for buildings that go beyond traditional efficiency measures.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Discrete fracture network model benchmarks developed and applied in a DECOVALEX-2023 repository performance assessment study

This study presents newly developed benchmarks for modeling flow and transport within discrete fracture networks (DFNs) and useful methods for analyzing the results. The new benchmarks are designed to test modeling approaches for use in probabilistic performance assessment models of deep geologic repositories in fractured rock. The benchmarks simulate flow and transport through a 1 km 3 block of fractured rock. The first simulates migration of a short pulse of tracer through a simple network of four intersecting fractures. The second adds 1089 stochastically generated fractures. The third changes the pulse to a continuous point source. Evaluation of model performance relies on moment analysis and comparison of the results of different models. The expected nondimensional first moment of the conservative tracer for each benchmark is 1. The benchmarks were simulated by teams from Canada, Czechia, Germany, Korea, Sweden, Taiwan, and the United States as part of a DECOVALEX-2023 study (decovalex.org). The teams used various approaches, including explicit DFN modeling, DFN upscaling to an equivalent continuous porous medium (ECPM), and a combination of both methods. Transport mechanisms are modeled using either the advection-dispersion equation or particle tracking. Results demonstrate strong agreement among the models in breakthrough behavior up to the 75th percentile. Significant deviations in first moments and well-clustered outputs led to the identification of inaccuracies in several models. Such findings exemplify the benefit of exercising these benchmarks and using the presented methods to test DFN flow and transport models.

Benchmark↗

A Comparison of Evidence-Based Estimates and Empirical Benchmarks of the Appropriate Rate of Use of Radiation Therapy in Ontario

Purpose: Estimates of the appropriate rate of use of radiation therapy (RT) are required for planning and monitoring access to RT. Our objective was to compare estimates of the appropriate rate of use of RT derived from mathematical models, with the rate observed in a population of patients with optimal access to RT. Methods and Materials: The rate of use of RT within 1 year of diagnosis (RT{sub 1Y}) was measured in the 134,541 cases diagnosed in Ontario between November 2009 and October 2011. The lifetime rate of use of RT (RT{sub LIFETIME}) was estimated by the multicohort utilization table method. Poisson regression was used to evaluate potential barriers to access to RT and to identify a benchmark subpopulation with unimpeded access to RT. Rates of use of RT were measured in the benchmark subpopulation and compared with published evidence-based estimates of the appropriate rates. Results: The benchmark rate for RT{sub 1Y}, observed under conditions of optimal access, was 33.6% (95% confidence interval [CI], 33.0%-34.1%), and the benchmark for RT{sub LIFETIME} was 41.5% (95% CI, 41.2%-42.0%). Benchmarks for RT{sub LIFETIME} for 4 of 5 selected sites and for all cancers combined were significantly lower than the corresponding evidence-based estimates. Australian and Canadian evidence-based estimates of RT{sub LIFETIME} for 5 selected sites differed widely. RT{sub LIFETIME} in the overall population of Ontario was just 7.9% short of the benchmark but 20.9% short of the Australian evidence-based estimate of the appropriate rate. Conclusions: Evidence-based estimates of the appropriate lifetime rate of use of RT may overestimate the need for RT in Ontario.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Benchmark probabilistic solar forecasts: Characteristics and recommendations

We illustrate and compare commonly used benchmark, or reference, methods for probabilistic solar forecasting that researchers use to measure the performance of their proposed techniques. A thorough review of the literature indicates wide variation in the benchmarks implemented in probabilistic solar forecast studies. To promote consistent and sensible methodological comparisons, we implement and compare ten variants from six common benchmark classes at two temporal scales: intra-hourly forecasts and hourly resolution forecasts. Using open-source Surface Radiation Budget Network (SURFRAD) data from 2018, these benchmark methods are compared using proper probabilistic metrics and common diagnostic tools. Practical implementation issues, such as the impact of missing data and applicability for operational forecasting, are also discussed. Furthermore, we make recommendations for practitioners on the appropriate selection of benchmark methods to properly showcase state-of-the-art improvements in forecast reliability and sharpness. All code and open-source data are available on Github for reproducibility and for other researchers to apply the same benchmark methods to their own data.

14 SOLAR ENERGY↗

Commissioning and benchmarking a 3D dosimetry system for clinical use

Purpose: A 3D dosimetry system is described which consists of two parts: a radiochromic plastic dosimeter PRESAGE (which responds to absorbed dose with a linear change in optical-density) and the Duke large-field-of-view optical-CT scanner (DLOS). The DLOS/PRESAGE system has recently been commissioned and benchmarked for clinical use and, in particular, for verification and commissioning of complex radiation treatments. Methods: DLOS commissioning involved determining the dynamic range, spatial resolution, noise, temporal, and other characteristics of the light source and imaging components. Benchmarking tests were performed on the combined DLOS/PRESAGE system to establish baseline dosimetric performance. The tests consisted of delivering simple radiation treatments to PRESAGE dosimeters, and comparing the measured 3D relative dose distributions with the known gold standard. The gold standard distribution was obtained from machine beam-data or the treatment planning system (TPS). All studies used standardized procedures to ensure consistency. Results: For commissioning, isotropic spatial resolution was submillimeter (MTF > 0.5 for frequencies of 1.5 lp/mm) and the dynamic range was {approx}60 dB. Flood field uniformity was within 10% and stable after 45 min of warm-up. Stray-light is small, due to telecentricity, but even the residual can be removed through deconvolution by a point-spread-function. For benchmarking, the mean 3D passing NDD (normalized dose distribution) rate (3%, 3mm, 5% dose threshold) over the benchmark data sets was 97.3% {+-} 0.6% (range 96%-98%), which is on par with other planar dosimeters used in external beam radiation therapy indicating excellent agreement. Noise was low at <2% of maximum dose (4-12 Gy) for 2 mm reconstructions. The telecentric design was critical to enabling fast imaging with minimal stray-light artifacts. Conclusions: This work presents the first comprehensive benchmarking of a 3D dosimetry system for clinical use. The DLOS/PRESAGE benchmark tests show consistently good agreement to simple known distributions. The system produces accurate isotropic 2 mm dose data over clinical volumes (e.g., 16 cm diameter phantoms, 12 cm height), in under 15 min. It represents a uniquely useful and versatile new tool for commissioning and verification of complex therapy treatments.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗