Results for “Statistical sampling techniques” — Page 2

DOE OSTI2024

A Provably Accurate Randomized Sampling Algorithm for Logistic Regression

In statistics and machine learning, logistic regression is a widely-used supervised learning technique primarily employed for binary classification tasks. When the number of observations greatly exceeds the number of predictor variables, we present a simple, randomized sampling-based algorithm for logistic regression problem that guarantees high-quality approximations to both the estimated probabilities and the overall discrepancy of the model. Our analysis builds upon two simple structural conditions that boil down to randomized matrix multiplication, a fundamental and well-understood primitive of randomized numerical linear algebra. We analyze the properties of estimated probabilities of logistic regression when leverage scores are used to sample observations, and prove that accurate approximations can be achieved with a sample whose size is much smaller than the total number of observations. To further validate our theoretical findings, we conduct comprehensive empirical evaluations. Overall, our work sheds light on the potential of using randomized sampling approaches to efficiently approximate the estimated probabilities in logistic regression, offering a practical and computationally efficient solution for large-scale datasets.

Chowdhury, Agniva↗

DOE OSTI2024

Graph Neural Networks for Parameterized Quantum Circuits Expressibility Estimation (Rev.1)

Parameterized quantum circuits (PQCs) are fundamental to quantum machine learning (QML), quantum optimization, and variational quantum algorithms (VQAs). The expressibility of PQCs is a measure that determines their capability to harness the full potential of the quantum state space. It is thus a crucial guidepost to know when selecting a particular PQC ansatz. However, the existing technique for expressibility computation through statistical estimation requires a large number of samples, which poses significant challenges due to time and computational resource constraints. This paper introduces a novel approach for expressibility estimation of PQCs using Graph Neural Networks (GNNs). We demonstrate the predictive power of our GNN model with a dataset consisting of 25,000 samples from the noiseless IBM QASM Simulator and 12,000 samples from three distinct noisy quantum backends. The model accurately estimates expressibility, with root mean square errors (RMSE) of 0.05 and 0.06 for the noiseless and noisy backends, respectively. We compare our model’s predictions with reference circuits from Sim et al. and IBM Qiskit’s hardwareefficient ansatz sets to further evaluate our model’s performance. Our experimental evaluation in noiseless and noisy scenarios reveals a close alignment with ground truth expressibility values, highlighting the model’s efficacy. Moreover, our model exhibits promising extrapolation capabilities, predicting expressibility values with low RMSE for out-of-range qubit circuits trained solely on only up to 5-qubit circuit sets. This work thus provides a reliable means of efficiently evaluating the expressibility of diverse PQCs on noiseless simulators and hardware.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

Characterizing the Uncertainty of Measurement of Traceable Isotope Ratios with Bayesian Statistical Techniques

Analytical techniques such as multicollector—inductively coupled plasma—mass spectrometry (MC-ICP-MS) are routinely employed at SRNL, other National Laboratories, and in academia to determine the precise isotopic composition of diverse natural and anthropogenic samples (e.g., rocks and nuclear materials). Quantifying and reporting uncertainty in such analyses, while regularly performed, have a rigorous statistical foundation. The Guide to the Expression of Uncertainty in Measurement 4 (GUM) outlines conventional techniques used to assess such uncertainty. As the accessibility and speed of statistical computing increase, there is a need to modernize conventional techniques. For example, Supplement 1 to the 3rd to the GUM suggests the use of approximation methods as an updated approach to the GUM.

McLarty, Ellis C.↗

DOE OSTI2022

The Dark Energy Survey supernova program: cosmological biases from supernova photometric classification

ABSTRACT Cosmological analyses of samples of photometrically identified type Ia supernovae (SNe Ia) depend on understanding the effects of ‘contamination’ from core-collapse and peculiar SN Ia events. We employ a rigorous analysis using the photometric classifier SuperNNova on state-of-the-art simulations of SN samples to determine cosmological biases due to such ‘non-Ia’ contamination in the Dark Energy Survey (DES) 5-yr SN sample. Depending on the non-Ia SN models used in the SuperNNova training and testing samples, contamination ranges from 0.8 to 3.5 per cent, with a classification efficiency of 97.7–99.5 per cent. Using the Bayesian Estimation Applied to Multiple Species (BEAMS) framework and its extension BBC (‘BEAMS with Bias Correction’), we produce a redshift-binned Hubble diagram marginalized over contamination and corrected for selection effects, and use it to constrain the dark energy equation-of-state, w. Assuming a flat universe with Gaussian ΩM prior of 0.311 ± 0.010, we show that biases on w are <0.008 when using SuperNNova, with systematic uncertainties associated with contamination around 10 per cent of the statistical uncertainty on w for the DES-SN sample. An alternative approach of discarding contaminants using outlier rejection techniques (e.g. Chauvenet’s criterion) in place of SuperNNova leads to biases on w that are larger but still modest (0.015–0.03). Finally, we measure biases due to contamination on w0 and wa (assuming a flat universe), and find these to be <0.009 in w0 and <0.108 in wa, 5 to 10 times smaller than the statistical uncertainties for the DES-SN sample.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2025

Deep inference of simulated strong lenses in ground-based surveys

The large number of strong lenses discoverable in future astronomical surveys will likely enhance the value of strong gravitational lensing as a cosmic probe of dark energy and dark matter. However, leveraging the increased statistical power of such large samples will require further development of automated lens modeling techniques. We show that deep learning and simulation-based inference (SBI) methods produce informative and reliable estimates of parameter posteriors for strong lensing systems in ground-based surveys. We present the examination and comparison of two approaches to lens parameter estimation for strong galaxy-galaxy lenses — Neural Posterior Estimation (NPE) and Bayesian Neural Networks (BNNs). We perform inference on 1-, 5-, and 12-parameter lens models for ground-based imaging data that mimics the Dark Energy Survey (DES). We find that NPE outperforms BNNs, producing posterior distributions that are more accurate, precise, and well-calibrated for most parameters. For the 12-parameter NPE model, the calibration is consistently within <10% of optimal calibration for all parameters, while the BNN is rarely within 20% of optimal calibration for any of the parameters. Similarly, residuals for most of the parameters are smaller (by up to an order of magnitude) with the NPE model than the BNN model. This work takes important steps in the systematic comparison of methods for different levels of model complexity.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

DOE OSTI2025

The Surface-Topography Challenge: A Multi-Laboratory Benchmark Study to Advance the Characterization of Topography

Surface performance is critically influenced by topography in virtually all real-world applications. The current standard practice is to describe topography using one of a few industry-standard parameters. The most commonly reported number is Ra, the average absolute deviation of the height from the mean line (at some, not necessarily known or specified, lateral length scale). However, other parameters, particularly those that are scale-dependent, influence surface and interfacial properties; for example the local surface slope is critical for visual appearance, friction, and wear. The present Surface-Topography Challenge was launched to raise awareness for the need of a multi-scale description, but also to assess the reliability of different metrology techniques. In the resulting international collaborative effort, 153 scientists and engineers from 64 research groups and companies across 20 countries characterized statistically equivalent samples from two different surfaces: a “rough” and a “smooth” surface. The results of the 2088 measurements constitute the most comprehensive surface description ever compiled. We find wide disagreement across measurements and techniques when the lateral scale of the measurement is ignored. Consensus is established through scale-dependent parameters while removing data that violates an established resolution criterion and deviates from the majority measurements at each length scale. Our findings suggest best practices for characterizing and specifying topography. The public release of the accumulated data and presented analyses enables global reuse for further scientific investigation and benchmarking.

42 ENGINEERING↗

DOE OSTI2026

Measurements of three-flavor neutrino oscillations from a PISCES two-detector fit to the NOvA Experiment data

NOvA is a long-baseline neutrino oscillation experiment with two functionally identical detectors: a Near Detector (ND) at Fermilab, placed 1 km from the neutrino source, and a Far Detector (FD) located 810 km away from the ND in Minnesota. NOvA s primary physics goals are to measure the neutrino oscillation parameters $\theta_{23}$ and $\Delta m^2_{32}$ with high precision, determine the neutrino mass hierarchy, and constrain the value of $\delta_{CP}$, primarily via the study of muon neutrino to electron neutrino oscillation. Extracting values for oscillation parameters from fits to data usually relies on treating systematic uncertainties as nuisance parameters, a strategy that suffers from poor scalability as the number of uncertainties becomes larger. This work introduces PISCES (Parameter Inference with Systematic Covariance and Exact Statistics), a novel method that circumvents this scalability problem by encoding systematic uncertainties into a covariance matrix. PISCES utilizes a nested minimization in which optimal systematic pulls are first computed using the covariance matrix in an inner minimization step, then the oscillation parameters are profiled over in the outer minimization. PISCES also uses a Poisson Likelihood term, making it ideal for the inclusion of low-statistic samples in the fits. PISCES is a flexible framework that also supports complex fits, such as a joint Near and Far detector fit. In the standard NOvA analysis, oscillation parameters are extracted using an extrapolation technique in which the ND data indirectly constrain the FD prediction via a ratio method. PISCES, on the other hand, enables a simultaneous ND+FD fit, allowing the high-statistics ND data to directly constrain systematic uncertainties across all samples. This thesis presents the full PISCES joint ND+FD fit for the NOvA three-flavor analysis, details its implementation, and evaluates its performance through extensive robustness tests and fake data studies. It also provides a comparison between the PISCES joint ND+FD results and the standard NOvA extrapolation method using the full NOvA 10-year data set. The results demonstrate that PISCES can successfully fit NOvA data while incorporating the constraints from the ND detectors consistently, using physically motivated systematic uncertainties to account for data/MC discrepancies.

Rajaoalisoa, Miriama [Cincinnati U.]↗

DOE OSTI2023

Multislice forward modeling of coherent surface scattering imaging on surface and interfacial structures

To study nanostructures on substrates, surface-sensitive reflection-geometry scattering techniques such as grazing incident small angle X-ray scattering are commonly used to yield an averaged statistical structural information of the surface sample. Grazing incidence geometry can probe the absolute three-dimensional structural morphology of the sample if a highly coherent beam is used. Coherent surface scattering imaging (CSSI) is a powerful yet non-invasive technique similar to coherent X-ray diffractive imaging (CDI) but performed at small angles and grazing-incidence reflection geometry. A challenge with CSSI is that conventional CDI reconstruction techniques cannot be directly applied to CSSI because the Fourier-transform-based forward models cannot reproduce the dynamical scattering phenomenon near the critical angle of total external reflection of the substrate-supported samples. To overcome this challenge, we have developed a multislice forward model which can successfully simulate the dynamical or multi-beam scattering generated from surface structures and the underlying substrate. The forward model is also demonstrated to be able to reconstruct an elongated 3D pattern from a single shot scattering image in the CSSI geometry through fast-performing CUDA-assisted PyTorch optimization with automatic differentiation.

77 NANOSCIENCE AND NANOTECHNOLOGY↗

DOE OSTI2022

Evolving Metrics for Resource Adequacy Assessment

Resource adequacy analysis quantifies the likelihood of capacity shortfall on a power system in a probabilistic manner. Using a combination of statistical techniques and power system fundamentals, the analysis typically evaluates hundreds or thousands of stochastic random samples (replications) of varying load, generator outages, variable renewable energy availability, and other aspects of power system uncertainty. In this range of uncertainty, there are - at times - periods where the power system's available resources are insufficient to meet system demand, referred to as a shortfall event. Today's power systems' rapidly evolving generation mix is changing the types of data needed by system planners and regulators, which can often render traditional resource adequacy metrics insufficient for ensuring resource adequacy for tomorrow's grid. In this paper we provide a critical assessment of traditional measures of shortfall risk in power systems, discussing their shortcomings and how they compare to metrics used in other domains. From this analysis we propose four steps forward for improving power system resource adequacy risk metrics in the future.

ENERGY PLANNING, POLICY, AND ECONOMY,POWER TRANSMI↗

DOE OSTI2023

Beyond Expected Values Evolving Metrics for Resource Adequacy Assessment

Resource adequacy analysis quantifies the likelihood of capacity shortfall on a power system in a probabilistic manner. Using a combination of statistical techniques and power system fundamentals, the analysis typically evaluates hundreds or thousands of stochastic random samples (replications) of varying load, generator outages, variable renewable energy availability, and other aspects of power system uncertainty. In this range of uncertainty, there are - at times - periods where the power system's available resources are insufficient to meet system demand, referred to as a shortfall event. Today's power systems' rapidly evolving generation mix is changing the types of data needed by system planners and regulators, which can often render traditional resource adequacy metrics insufficient for ensuring resource adequacy for tomorrow's grid. In this paper we provide a critical assessment of traditional measures of shortfall risk in power systems, discussing their shortcomings and how they compare to metrics used in other domains. From this analysis we propose four steps forward for improving power system resource adequacy risk metrics in the future.

ENERGY PLANNING, POLICY, AND ECONOMY↗

DOE OSTI2026

An Integrated Framework for Memory-Centric Analysis: From Trace Collection to Co-Design

The memory wall phenomenon—where advances in processor performance significantly outpace those in memory subsystems—poses a fundamental challenge for contemporary computing systems. In memory-bound applications, memory subsystem behavior dominates performance, yet existing analysis approaches present significant limitations: detailed microarchitectural simulators require days to weeks to simulate modest workloads; hardware performance counters provide only aggregate statistics that obscure temporal and spatial access patterns; and scaled simulation approaches face challenges in capturing certain behaviors that emerge at larger scales. These limitations reflect a processor-centric design philosophy increasingly misaligned with memory-bound workloads where detailed understanding of memory access patterns, cache hierarchy interactions, and contention is critical for effective optimization. This paper presents an integrated framework for memory-centric analysis that enables effective hardware-software co-design. We describe practical trace collection techniques, including hardware-assisted processor tracing with minimal overhead and portable software-based instrumentation with statistical sampling. We present multi-perspective analysis methods that examine memory behavior from temporal, sequential, spatial, and relational viewpoints, revealing distinct optimization opportunities invisible in aggregate metrics. We detail an architectural modeling framework that uses sampled traces with temporal interpolation and confidence-based filtering to evaluate cache and memory configurations. Evaluation on representative benchmarks demonstrates that this framework achieves practical accuracy (L2 cache errors of 2.64\%, confidence-filtered L3 errors of 9.92\%, bandwidth errors of 7.33\%) while providing substantial speedup (26.8×) over cycle-accurate simulation, enabling rapid design space exploration. We demonstrate how this integrated framework enables systematic identification of both hardware optimizations (memory controller tuning, bank partitioning, NUMA configuration) and software optimizations (data layout restructuring, prefetching strategies, memory-aware scheduling). Through this comprehensive treatment of the memory-centric analysis pipeline—from trace collection through architectural modeling to co-design application—we provide researchers and practitioners with practical techniques for addressing memory bottlenecks in contemporary computing systems.

Gajaria, Dhruv Mayur↗

DOE OSTI2022

High-Energy X-ray Diffraction Microscopy for Nuclear Forensics FY2022 Project Report

Morphological information on nuclear material has been identified using visible light and scanning electron microscopy. These identify qualitative differences in particle morphology. Three-dimensional imaging of materials through alternating scanning electron microscopy imaging and focused ion beam milling has also been used. Unfortunately, these techniques are time- and labor-intensive, with significant sample preparation required and lengthy analysis times. Further, the resulting 3D images are qualitative, require manual identification, and do not capture statistically-representative populations. High energy X-ray 3D imaging using a direct-beam or diffracted-beam (High-Energy Diffraction Microscopy) have been developed at the Advanced Photon Source and can produce quantitative information on grains (phase, location, etc.) and pores (size distribution, sphericity) in a material. These techniques require only minutes to characterize a sample volume and are non-destructive, thus suitable for a wide range of existing samples and for confirmatory analyses to be carried out using conventional microscopy techniques. In this first year of the project, all uranium oxide samples were synthesized and characterized using conventional analyses by the analytical chemistry laboratory. Conventional analysis methods included powder x-ray diffraction, scanning electron microscopy, impurity analysis via inductively coupled plasma mass spectrometry, and infrared spectroscopy. Impurity analysis shows a drop in boron content from UO 3 to the lowest U 3 O 8 calcination temperature, but otherwise no appreciable difference in any sample. Analysis of diffraction data shows a flip of peaks from UO 3 dominated for the 600 °C calcined sample to U 3 O 8 dominated at 700 °C and 800 °C. Analysis of scanning electron microscopy images shows that with increased calcination temperature the size distribution of particles seems to increase and broaden. Both of these last findings are in line with previously published data, though this work used significantly fewer particles to simply show similar trends instead of getting truly quantitative particle analysis. Infrared analysis similarly shows ingrowth of U 3 O 8 as calcination temperature is increased, along with depression of peaks associated with UO 3 and water. Samples were prepared for analysis at the Advanced Photon Source at beamline 1-ID. It is anticipated that analysis will occur in November of 2022. AI/ML techniques to de-noise data coming out of 1-ID during the analyses was also developed during this time using previously gathered data. Preliminary results using a self-supervision technique called Noise2Selfshow good de-noising of data. Once the uranium oxide samples are analyzed, real data will be used to test the de-noising and other AI/ML techniques that may be developed in the second year of the project.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

DOE OSTI2023

High-Energy X-ray Diffraction Microscopy for Nuclear Forensics (FY23 Project Report)

Morphological information on nuclear material has been identified using visible light and scanning electron microscopy. These identify qualitative differences in particle morphology. Three-dimensional imaging of materials through alternating scanning electron microscopy imaging and focused ion beam milling has also been used. Unfortunately, these techniques are time- and labor-intensive, with significant sample preparation required and lengthy analysis times. Further, the resulting 3D images are qualitative, require manual identification, and do not capture statistically-representative populations. High energy X-ray 3D imaging using a direct-beam or diffracted-beam (High-Energy Diffraction Microscopy) have been developed at the Advanced Photon Source and can produce quantitative information on grains (phase, location, etc.) and pores (size distribution, sphericity) in a material. These techniques require only minutes to characterize a sample volume and are non-destructive, thus suitable for a wide range of existing samples and for confirmatory analyses to be carried out using conventional microscopy techniques.

36 MATERIALS SCIENCE↗

DOE OSTI2024

Measuring Muon Antineutrino Charged-Current Interactions Without Mesons in the Final State, in the NOvA Near Detector

NOvA is a long-baseline neutrino experiment based at Fermilab in the US, with the primary aim of measuring neutrino and antineutrino oscillations. This will enhance our understanding of electroweak interactions by measuring the neutrino mixing angles, CP-violating phase and neutrino mass ordering. To measure these oscillations, we first need to have a deep understanding of how neutrinos and antineutrinos interact with matter. Antineutrino interaction cross sections are, at present, particularly poorly constrained, and processes such as meson exchange currents are not well understood in the antineutrino sector. This analysis will develop a cross-section measurement of muon antineutrino interactions without mesons (e.g. pions or kaons) in the final state, in the NOvA near detector. A high-statistics, high-purity sample is obtained through a cut-based selection process implementing machine learning techniques. The sample is dominated by quasi-elastic and meson exchange current interactions which are sensitive to nuclear effects such as Final-State Interactions. The cross section will be extracted as a function of the incoming neutrino energy and the kinematics of the outgoing particles.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2024

Measuring Muon Antineutrino Charged-Current Interactions without Mesons in the Final State, in the NOvA Near Detector

NOvA is a long-baseline neutrino experiment based at Fermilab in the US, with the primary aim of measuring neutrino and antineutrino oscillations. This will enhance our understanding of electroweak interactions by measuring the neutrino mixing angles, CP-violating phase and neutrino mass ordering. To measure these oscillations, we first need to have a deep understanding of how neutrinos and antineutrinos interact with matter. Antineutrino interaction cross sections are, at present, particularly poorly constrained, and processes such as meson exchange currents are not well understood in the antineutrino sector. This analysis will develop a cross-section measurement of muon antineutrino interactions without mesons (e.g. pions or kaons) in the final state, in the NOvA near detector. A high-statistics, high-purity sample is obtained through a cut-based selection process implementing machine learning techniques. The sample is dominated by quasi-elastic and meson exchange current interactions which are sensitive to nuclear effects such as Final-State Interactions. The cross section will be extracted as a function of the incoming neutrino energy and the kinematics of the outgoing particles. This presentation will give an overview of the analysis and discuss progress towards obtaining the cross-section measurement.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2026

Transformative Impacts of Laser-Induced Breakdown Spectroscopy on Environmental and Biological Research at Oak Ridge National Laboratory

This manuscript will present an advancement of transformative research that has been conducted at Oak Ridge National Laboratory (ORNL) over a 25-year period (2000–2025) on a variety of environmental and biological matrices. These investigations derived a fundamental understanding of how elemental detection and analysis of these matrices led to the knowledge and discovery of natural processes in plants and the environment. Each project led to the initiation of a new research area which unearthed awesome and novel breakthroughs. Highlights are listed below: 1. The preliminary research at ORNL centered on the detection of aerosols utilizing Laser-induced Breakdown Spectroscopy (LIBS) technology. The Clean Air Act Amendment (CAAA) of 1990 highlighted the importance of identifying hazardous air pollutants (HAPs) due to their impact on environmental and human health, thereby underscoring the need to detect various toxic elements. Research in aerosol chemistry aimed to identify these harmful elements released by factories during periods of increased emissions in their manufacturing processes. LIBS emerged as the most effective method for real-time, in situ measurements of metal species in both gaseous and aerosol phases. 2. An understanding of the presence of total carbon in soils gives perspective on how to develop carbon sequestration strategies. The recognition that carbon sinks can evolve back to carbon sources to emit back to the atmosphere was an important consideration. Also, the concentration of carbon in soil indicates the health of land areas for growing crops successfully. 3. The direct detection of most of the elements in a wood sample in a single emission spectrum, without sample preparation, encouraged the research to use the LIBS technique for preservative treated wood coupled with use of multivariate statistical methodology. Additionally, it encouraged the researchers to try to differentiate natural woods from different parts of the country, and it was successfully demonstrated that LIBS coupled with MVA analysis could differentiate wood of different species from each other and of similar species grown in different environments based on their elemental spectra. This was a breakthrough since it revealed a systematic approach to connect elemental scarcity and abundance to either drought or typical rainfall conditions for the hardwood trees grown in specific areas. 4. Furthermore, the research progressed to reveal physiological and developmental processes contributing to biomass production such that the variation in leaf elemental composition increases our understanding of terrestrial nutrient cycles, as well as tracking the transfer of toxic elements from soils to living organisms. 5. Recently another breakthrough viz., ionomics initiated the correlation of elements to specific genes, uncovering the function that the element performed in the plant. More recently, this has been extended from plants to fungi as well as fungi growing in symbiotic relations with plants.

09 BIOMASS FUELS↗

DOE OSTI2022

Ensemble approximate control variate estimators: Applications to multi-fidelity importance sampling.

The recent growth in multifidelity uncertainty quantification has given rise to a large set of variance reduction techniques that leverage information from model ensembles to provide variance reduction for estimates of the statistics of a high-fidelity model. In this paper we provide two contributions: (1) we utilize an ensemble estimator to account for uncertainties in the optimal weights of approximate control variate (ACV) approaches and derive lower bounds on the number of samples required to guarantee variance reduction; and (2) we extend an existing multifidelity importance sampling (MFIS) scheme to leverage control variates. Our approach directly addresses a limitation of many multifidelity sampling strategies that require the usage of pilot samples to estimate covariances. As such we make significant progress towards both increasing the practicality of approximate control variates—for instance, by accounting for the effect of pilot samples—and using multifidelity approaches more effectively for estimating low-probability events. The numerical results indicate our hybrid MFIS-ACV estimator achieves up to 50% improvement in variance reduction over the existing state-of-the-art MFIS estimator, which had already shown an outstanding convergence rate compared to the Monte Carlo method, on several problems of computational mechanics.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2026

Enhanced climate reproducibility testing with false discovery rate correction

Simulating the Earth's climate is an important and complex problem, thus climate models are similarly complex, comprised of millions of lines of code. In order to appropriately utilize the latest computational and software infrastructure advancements in Earth system models running on modern hybrid computing architectures to improve their performance, precision, accuracy, or all three; it is important to ensure that model simulations are repeatable and robust. This introduces the need for establishing statistical or non-bit-for-bit reproducibility, since bit-for-bit reproducibility may not always be achievable. Here, we propose a short-simulation ensemble-based test for an atmosphere model to evaluate the null hypothesis that modified model results are statistically equivalent to that of the original model. We implement this test in version 2 of the US Department of Energy's Energy Exascale Earth System Model (E3SM). The test evaluates a standard set of output variables across the two simulation ensembles and uses a false discovery rate correction to account for multiple testing. The false positive rates of the test are examined using re-sampling techniques on large simulation ensembles and are found to be lower than the currently implemented bootstrapping-based testing approach in E3SM. We also evaluate the statistical power of the test using perturbed simulation ensemble suites, each with a progressively larger magnitude of change to a tuning parameter. The new test is generally found to exhibit more statistical power than the current approach, being able to detect smaller changes in parameter values with higher confidence.

Kelleher, Michael E. [Oak Ridge National Laborator↗