SEARCH · Engineering Papers

Results for “ensemble prediction”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 577 records · Page 32

All sources NASA NTRS DOE OSTI

DOE OSTI2023

Uncertainty quantification in machine learning for engineering design and health prognostics: A tutorial

On top of machine learning (ML) models, uncertainty quantification (UQ) functions as an essential layer of safety assurance that could lead to more principled decision making by enabling sound risk assessment and management. The safety and reliability improvement of ML models empowered by UQ has the potential to significantly facilitate the broad adoption of ML solutions in high-stakes decision settings, such as healthcare, manufacturing, and aviation, to name a few. In this tutorial, we aim to provide a holistic lens on emerging UQ methods for ML models with a particular focus on neural networks and the applications of these UQ methods in tackling engineering design as well as prognostics and health management problems. Towards this goal, we start with a comprehensive classification of uncertainty types, sources, and causes pertaining to UQ of ML models. Next, we provide a tutorial-style description of several state-of-the-art UQ methods: Gaussian process regression, Bayesian neural network, neural network ensemble, and deterministic UQ methods focusing on spectral-normalized neural Gaussian process. Established upon the mathematical formulations, we subsequently examine the soundness of these UQ methods quantitatively and qualitatively (by a toy regression example) to examine their strengths and shortcomings from different dimensions. Then, we review quantitative metrics commonly used to assess the quality of predictive uncertainty in classification and regression problems. Afterward, we discuss the increasingly important role of UQ of ML models in solving challenging problems in engineering design and health prognostics. In conclusion, two case studies with source codes available on GitHub are used to demonstrate these UQ methods and compare their performance in the life prediction of lithium-ion batteries at the early stage (case study 1) and the remaining useful life prediction of turbofan engines (case study 2).

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Accurate Kohn-Sham auxiliary system from the ground-state density of solids

The Kohn-Sham (KS) system is an auxiliary system whose effective potential is unknown in most cases. It is in principle determined by the ground-state density and it has been found numerically for some low-dimensional systems by inverting the KS equations starting from a given accurate density. For solids, only approximate results are available. In this work, we determine accurate exchange-correlation (xc) potentials for Si and NaCl using the ground-state densities obtained from auxiliary field quantum Monte Carlo calculations. We show that these xc potentials can be rationalized as an ensemble of a few local functions of the density, whose form depends on the specific environment and can be well characterized by the gradient of the density and the local kinetic energy density. Further, the KS band structure can be obtained with high accuracy. The true KS band gap turns out to be larger than the prediction of the local density approximation, but significantly smaller than the measurable photoemission gap, which confirms previous estimates. Finally, our findings show that the conjecture that very different xc potentials can lead to very similar densities and other KS observables is true also in solids, which questions the meaning of details of the potentials and, at the same time, confirms the stability of the KS system.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2020

A Novel Active Optimization Approach for Rapid and Efficient Design Space Exploration Using Ensemble Machine Learning

In this work, a novel design optimization technique based on active learning, which involves dynamic exploration and exploitation of the design space of interest using an ensemble of machine learning algorithms, is presented. In this approach, a hybrid methodology incorporating an explorative weak learner (regularized basis function model) that fits high-level information about the response surface and an exploitative strong learner (based on committee machine) that fits finer details around promising regions identified by the weak learner is employed. For each design iteration, an aristocratic approach is used to select a set of nominees, where points that meet a threshold merit value as predicted by the weak learner are selected for evaluation. In addition to these points, the global optimum as predicted by the strong learner is also evaluated to enable rapid convergence to the actual global optimum once the most promising region has been identified by the optimizer. Additionally, this methodology is first tested by applying it to the optimization of a two-dimensional multi-modal surface and, subsequently, to a complex internal combustion (IC) engine combustion optimization case with nine control parameters related to fuel injection, initial thermodynamic conditions, and in-cylinder flow. It is found that the new approach significantly lowers the number of function evaluations that are needed to reach the optimum design configuration (by up to 80%) when compared to conventional optimization techniques, such as particle swarm and genetic algorithm-based optimization techniques.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DOE OSTI2024

Smaller Is Better: The Case for Lower-Order Iodoplumbate Species Dominating MAPbI 3 /Dimethylformamide Solutions

Here, using complementary experimental measurements and computational predictions of spectroscopic measurements (EXAFS, XANES, and UV–vis), we have determined the identity of the most stable iodoplumbate species in dilute lead halide perovskite precursor solutions. We have determined which species are most likely to be thermodynamically stable compared to others that are unstable or metastable. Condensed phase ab initio models were constructed, and the resulting ensembles were used to directly compare the computed signals to the experimental results of the EXAFS, XANES, and UV–vis spectra of PbI 2 :MAI in DMF. The results of this study suggest that only Pb 2+ , PbI + , and PbI 2 are dominant in the dilute lead perovskite precursor solutions as thermodynamically stable entities. Our interpretation of the relative stability of iodoplumbate species in solution, based on an analysis of EXAFS and XANES spectra, provides critically important new insight into the species most likely to be responsible for crystal nucleation and growth in these materials. This insight will have a significant consequence on the broad scientific community and will necessitate the reinterpretation of peaks in the UV–vis spectra of lead halide perovskite precursor solutions.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2020

27 AlNMR chemical shift of Al(OH)$^{-}_{4}$ calculated from first principles: Assessment of error cancellation in chemically distinct reference and target systems

Predicting accurate nuclear magnetic resonance chemical shieldings relies upon cancellation of different types of errors between the theoretically calculated shielding constant of the analyte of interest and the reference. Often, the intrinsic error in computed shieldings due to basis sets, approximations in the Hamiltonian, description of the wave function, and dynamic effects is nearly identical between the analyte and reference, yet if the electronic structure or sensitivity to local environment differs dramatically, this cannot be taken for granted. Detailed prior work has examined the octahedral trivalent cation Al(H 2 O)$^{3+}_{6}$, accounting for ab initio intrinsic errors. However, the use of this species as a reference for the chemically distinct tetrahedral anion Al(OH)$^{-}_{4}$ requires an understanding of how these errors cancel, in order to define the limits of accurately predicting 27 Al chemical shielding in Al(OH)$^{-}_{4}$. In this work, we estimate the absolute shielding of the 27 Al nucleus in Al(OH)$^{-}_{4}$ at the coupled cluster level (515.1± 5.3 ppm). Shielding sensitivity to the choice of method approximation and atomic basis sets used has been evaluated. Solvent and thermal effects are assessed through ensemble averaging techniques using ab-initio molecular dynamics. The contribution of each type of intrinsic error is assessed for the Al(H 2 O)$^{3+}_{6}$ and Al(OH)$^{-}_{4}$ ions, revealing significant differences that fundamentally hamper the ability to accurately calculate the 27 Al chemical shift of Al(OH)$^{-}_{4}$ from first principles.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2021

Leveraging machine learning to improve understanding and predictability of weather/climate extremes and the resilience of human systems

Focal Area(s): Developing hybrid models for predicting weather/climate extremes at the intersection of the urban environment (focal area #2). Science Challenge: Limited by many sources of uncertainty, numerical models are unable to accurately simulate weather and climate processes and this problem is particularly acute for modeling extreme events, including hurricanes. Moreover, computational limits within these numerical models bound our ability to develop large ensembles to explore this uncertainty and the implications relative to characterization of risk and resilience within the human systems that are impacted by changes in extremes.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2022

Distilling physical origins of hardness in multi-principal element alloys directly from ensemble neural network models

Abstract Despite a plethora of data being generated on the mechanical behavior of multi-principal element alloys, a systematic assessment remains inaccessible via Edisonian approaches. We approach this challenge by considering the specific case of alloy hardness, and present a machine-learning framework that captures the essential physical features contributing to hardness and allows high-throughput exploration of multi-dimensional compositional space. The model, tested on diverse datasets, was used to explore and successfully predict hardness in Al x Ti y (CrFeNi) 1- x - y , Hf x Co y (CrFeNi) 1- x - y and Al x (TiZrHf) 1- x systems supported by data from density-functional theory predicted phase stability and ordering behavior. The experimental validation of hardness was done on TiZrHfAl x . The selected systems pose diverse challenges due to the presence of ordering and clustering pairs, as well as vacancy-stabilized novel structures. We also present a detailed model analysis that integrates local partial-dependencies with a compositional-stimulus and model-response study to derive material-specific insights from the decision-making process.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2024

Models, data, and scripts associated with “Prediction of Distributed River Sediment Respiration Rates using Community-Generated Data and Machine Learning”

This data package is associated with the publication “Prediction of Distributed River Sediment Respiration Rates using Community-Generated Data and Machine Learning’’ submitted to the Journal of Geophysical Research: Machine Learning and Computation (Scheibe et al. 2024). River sediment respiration observations are expensive and labor intensive to obtain and there is no physical model for predicting this quantity. The Worldwide Hydrobiogeochemisty Observation Network for Dynamic River Systems (WHONDRS) observational data set (Goldman et al.; 2020) is used to train machine learning (ML) models to predict respiration rates at unsampled sites. This repository archives training data, ML models, predictions, and model evaluation results for the purposes of reproducibility of the results in the associated manuscript and community reuse of the ML models trained in this project. One of the key challenges in this work was to find an optimum configuration for machine learning models to work with this feature-rich (i.e. 100+ possible input variables) data set. Here, we used a two-tiered approach to managing the analysis of this complex data set: 1) a stacked ensemble of ML models that can automatically optimize hyperparameters to accelerate the process of model selection and tuning and 2) feature permutation importance to iteratively select the most important features (i.e. inputs) to the ML models. The major elements of this ML workflow are modular, portable, open, and cloud-based, thus making this implementation a potential template for other applications. This data package is associated with the GitHub repository found at Please see the file level metadata (flmd; “sl-archive-whondrs_flmd.csv”) for a list of all files contained in this data package and descriptions for each. Please see the data dictionary (dd; “sl-archive-whondrs_dd.csv”) for a list of all column headers contained within comma separated value (csv) files in this data package and descriptions for each. The GitHub repository is organized into five top-level directories: (1) “input_data” holds the training data for the ML models; (2) “ml_models” holds machine learning models trained on the data in “input_data”; (3) “scripts” contains data preprocessing and postprocessing scripts and intermediate results specific to this data set that bookend the ML workflow; (4) “examples” contains the visualization of the results in this repository including plotting scripts for the manuscript (e.g., model evaluation, FPI results) and scripts for running predictions with the ML models (i.e., reusing the trained ML models); (5) “output_data” holds the overall results of the ML model on that branch. Each trained ML model resides on its own branch in the repository; this means that inputs and outputs can be different branch-to-branch. Furthermore, depending on the number of features used to train the ML models, the preprocessing and postprocessing scripts, and their intermediate results, can also be different branch-to-branch. The “main-*” branches are meant to be starting points (i.e. trunks) for each model branch (i.e. sprouts). Please see the Branch Navigation section in the top-level README.md in the GitHub repository for more details. There is also one hidden directory “.github/workflows”. This hidden directory contains information for how to run the ML workflow as an end-to-end automated GitHub Action but it is not needed for reusing the ML models archived here. Please the top-level README.md in the GitHub repository for more details on the automation.

13C↗

DOE OSTI2025

Lattice QCD Study of Pion Electroproduction and Weak Production from a Nucleon

Quantum fluctuations in QCD influence nucleon structure and interactions, with pion production serving as a key probe of chiral dynamics. In this Letter, we present a lattice QCD calculation of multipole amplitudes at threshold, related to both pion electroproduction and weak production from a nucleon, using two gauge ensembles near the physical pion mass. We develop a technique for spin projection and construct multiple operators for analyzing the generalized eigenvalue problem in both the nucleon-pion system in the center-of-mass frame and the nucleon system with nonzero momentum. The numerical lattice results are then compared with those extracted from experimental data and predicted by low-energy theorems incorporating one-loop corrections. Published by the American Physical Society 2025

Gao, Yu-Sheng (ORCID:0009000406829247)↗

DOE OSTI2025

Structural switching dynamically controls the doubly pseudoknotted Rous sarcoma virus–programmed ribosomal frameshifting element

A hallmark of retrovirus replication is the translation of two different polyproteins from one RNA through programmed –1 frameshifting. This is a mechanism in which the actively translating ribosome is induced to slip in the 5′ direction at a defined codon and then continues translating in the new reading frame. Programmed frameshifting controls the stoichiometry of viral proteins and is therefore under stringent evolutionary selection. Forty years ago, the first frameshifting stimulatory element was discovered in the Rous sarcoma virus. The ~120 nt RNA segment was predicted to contain a pseudoknot, but its 3D structure has remained elusive. Now, we have determined cryoEM and X-ray crystallographic structures of this classic retroviral element, finding that it adopts a butterfly-like double-pseudoknot fold. One “wing” contains a dynamic pyrimidine-rich helix, observed crystallographically in two conformations and in a third conformation via cryoEM. The other wing encompasses the predicted pseudoknot, which interacts with a second unexpected pseudoknot through a toggle residue, A2546. This key purine switches conformations between structural states and tunes the stability of interacting residues in the two wings. We find that its mutation can modulate frameshifting by as much as 50-fold, likely by altering the relative abundance of different structural states in the conformational ensemble of the RNA. Taken together, our structure–function analyses reveal how a dynamic double pseudoknot junction stimulates frameshifting by taking advantage of conformational heterogeneity, supporting a multistate model in which high Shannon entropy enhances frameshifting efficiency.

Science & Technology - Other Topics↗

DOE OSTI2026

The AWAKEN wind farm benchmark, Part 2: Modeling results

Accurately modeling wind farm performance in complex atmospheric flows remains a challenge. This paper presents the modeling results of the American WAKE experimeNt (AWAKEN) wind farm benchmark, a collaborative effort involving 16 research groups from academia and industry within the International Energy Agency Wind Technology Collaboration Programme Task 57. The study evaluates a diverse suite of simulation tools, ranging from fast-running engineering wake models to high-fidelity large-eddy simulations, against a diurnal case study observed during the AWAKEN campaign. The benchmark utilized a three-phase structure to progressively assess model performance as observational data availability increased. Initial blind predictions showed that higher-fidelity models did not uniformly outperform simpler simulation tools. A distinct spatial bias was observed where models struggled to resolve the interplay between a low-level jet, wakes, and terrain-induced flow acceleration. In subsequent phases, leveraging additional measurements for model improvement led to a reduction in mean absolute error across the model ensemble; however, this effect was most pronounced in engineering wake models, where targeted calibration reduced error by up to 40~\%. Overall, the study demonstrates that inflow characterization remains a primary prerequisite for accuracy, particularly for models relying on coarse forcing datasets. While the limited ability to resolve local terrain-flow interactions under single-day conditions represent a recognized constraint, the overall findings on wake modeling and real-world validation still provide valuable guidance for model application and for mitigating this limitation.

Bodini, Nicola↗

DOE OSTI2024

Evaluation of FluSight influenza forecasting in the 2021–22 and 2022–23 seasons with a new target laboratory-confirmed influenza hospitalizations

Accurate forecasts can enable more effective public health responses during seasonal influenza epidemics. For the 2021–22 and 2022–23 influenza seasons, 26 forecasting teams provided national and jurisdiction-specific probabilistic predictions of weekly confirmed influenza hospital admissions for one-to-four weeks ahead. Forecast skill is evaluated using the Weighted Interval Score (WIS), relative WIS, and coverage. Six out of 23 models outperform the baseline model across forecast weeks and locations in 2021–22 and 12 out of 18 models in 2022–23. Averaging across all forecast targets, the FluSight ensemble is the 2nd most accurate model measured by WIS in 2021–22 and the 5th most accurate in the 2022–23 season. Forecast skill and 95% coverage for the FluSight ensemble and most component models degrade over longer forecast horizons. In this work we demonstrate that while the FluSight ensemble was a robust predictor, even ensembles face challenges during periods of rapid change.

59 BASIC BIOLOGICAL SCIENCES↗

DOE OSTI2021

Deep learning of accurate force field of ferroelectric HfO 2

The discovery of ferroelectricity in HfO 2 -based thin films opens up new opportunities for using this silicon-compatible ferroelectric to realize low-power logic circuits and high-density nonvolatile memories. The functional performances of ferroelectrics are intimately related to their dynamic responses to external stimuli such as electric fields at finite temperatures. Molecular dynamics is an ideal technique for investigating dynamical processes on large length and time scales, though its applications to new materials are often hindered by the limited availability and accuracy of classical force fields. Here we present a deep neural network–based interatomic force field of HfO 2 learned from ab initio data using a concurrent learning procedure. The model potential is able to predict structural properties such as elastic constants, equation of states, phonon dispersion relationships, and phase transition barriers of various hafnia polymorphs with accuracy comparable with density functional theory calculations. Here, the validity of this model potential is further confirmed by the reproduction of experimental sequences of temperature-driven ferroelectric-paraelectric phase transitions of HfO 2 with isobaric-isothermal ensemble molecular dynamics simulations. We suggest a general approach to extend the model potential of HfO 2 to related material systems including dopants and defects.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2023

Quantification of hydraulic trait control on plant hydrodynamics and risk of hydraulic failure within a demographic structured vegetation model in a tropical forest (FATES–HYDRO V1.0)

Abstract. Vegetation plays a key role in the global carbon cycle and thus is an important component within Earth system models (ESMs) that project future climate. Many ESMs are adopting methods to resolve plant size and ecosystem disturbance history, using vegetation demographic models. These models make it feasible to conduct more realistic simulation of processes that control vegetation dynamics. Meanwhile, increasing understanding of the processes governing plant water use, and ecosystem responses to drought in particular, has led to the adoption of dynamic plant water transport (i.e., hydrodynamic) schemes within ESMs. However, the extent to which variations in plant hydraulic traits affect both plant water stress and the risk of mortality in trait-diverse tropical forests is understudied. In this study, we report on a sensitivity analysis of an existing hydrodynamic scheme (HYDRO) model that is updated and incorporated into the Functionally Assembled Terrestrial Ecosystem Simulator (FATES) (FATES–HYDRO V1.0). The size- and canopy-structured representation within FATES is able to simulate how plant size and hydraulic traits affect vegetation dynamics and carbon–water fluxes. To better understand this new model system, and its functionality in tropical forest systems in particular, we conducted a global parameter sensitivity analysis at Barro Colorado Island, Panama. We assembled 942 observations of plant hydraulic traits on 306 tropical plant species for stomata, leaves, stems, and roots and determined the best-fit statistical distribution for each trait, which was used in model parameter sampling to assess the parametric sensitivity. We showed that, for simulated leaf water potential and loss of hydraulic conductivity across different plant organs, the four most important traits were associated with xylem conduit taper (buffers increasing hydraulic resistance with tree height), stomatal sensitivity to leaf water potential, maximum stem hydraulic conductivity, and the partitioning of total hydraulic resistance above vs. belowground. Our analysis of individual ensemble members revealed that trees at a high risk of hydraulic failure and potential tree mortality generally have a lower conduit taper, lower maximum xylem conductivity, lower stomatal sensitivity to leaf water potential, and lower resistance to xylem embolism for stem and transporting roots. We expect that our results will provide guidance on future modeling studies using plant hydrodynamic models to predict the forest responses to droughts and future field campaigns that aim to better parameterize plant hydrodynamic models.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2020

Nanoparticle Tracking to Probe Transport in Porous Media

From the granular and fractured subsurface environment to highly engineered polymer membranes used in pharmaceutical purification, porous materials are ubiquitous in nature and industrial applications. In particular, porous media are used extensively in processes including water treatment, pharmaceutical sterilization, food/beverage processing and heterogeneous catalysis, where hindered mass transport is either essential to the process or a necessary, but undesirable limitation. Unfortunately, there are currently no universal models capable of predicting mass transport based on a description of the porous material because real porous materials are complex, and because many coupled dynamic mechanisms (e.g. adsorption, steric effects, hydrodynamic effects, electrostatic interactions, etc.) give rise to the observed macroscopic transport phenomena. While classical techniques, like nuclear magnetic resonance and dynamic light scattering, provide useful information about mass transport in porous media at the ensemble level, they provide limited insight into the microscopic mechanisms that give rise to complex phenomena such as anomalous diffusion, hindered pore-space accessibility, and unexpected retention under flow, among many others. To address this issue, we have developed refractive index matching imaging systems, combined with single-particle tracking methods, allowing the direct visualization of single particle motion within a variety of porous materials. Here, we summarize our recent efforts to advance the understanding of nanoparticle transport in porous media using single-particle tracking methods in both fundamental and applied scenarios. First, we describe the basic principles for two-dimensional and three-dimensional single-particle tracking in porous materials. Then, we describe concrete examples of nanoparticle transport in porous materials from two perspectives: (1) understanding fundamental elementary particle transport processes in porous media, including pore accessibility and cavity escape, which limit transport in porous media; (2) facilitating applications in industrial processes, e.g. by understanding the mechanisms of particle fouling and remobilization in filtration membranes. Finally, we provide an outlook of opportunities associated with investigating other types of mass transport in confined environments using single-particle tracking methods, including electrophoretic and self-propelled motion.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2025

Detection of Diversion in a Realistic Heat Pipe Microreactor Using Supervised Machine Learning

Microreactors (MRs) pose new challenges for international safeguards. Here, their small size and mass reproducibility make them ideal for deployment in greater numbers and in remote locations, making the job of safeguards inspectors more challenging. Machine learning (ML) is currently being applied to many fields to augment human performance and increase automation; in particular, ML could be used to provide insight for international inspectors to help detect the diversion of nuclear fuel from MR cores. Four ML model types (k-nearest neighbors, decision tree, random forest, and histogram-based gradient boosted ensemble) were trained on integrated flux and critical control drum angle data generated with Serpent 2 for a realistic heat pipe MR design, achieving nearly 100% binary classification accuracy of nominal and diversion core configurations by the end of 1 full power year for three of the four model types. Regression model variants were also trained, using the same input data, for predicting the number of fuel pins diverted. Root-mean-square errors below 5% of the total number of fuel pins were achieved by the 1 full power year mark for all models.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

DOE OSTI2025

Establishing model credibility for process-microstructure-property relationships in additive manufacturing using exascale computing

Additive Manufacturing (AM) of alloys holds significant promise as a disruptive technology in various industries, yet its adoption is often hindered by challenges in achieving consistent part quality. These issues are primarily due to the complex process-microstructure-property (PSP) relationships inherent to AM. Computational models can greatly aid in understanding these relationships, but their widespread impact and adoption has been limited by a lack of validated, open-source, and computationally efficient PSP modeling frameworks and hardware limitations. Here, this study leverages the ExaAM software suite and data from the AMBench-2018 series of laser powder bed fusion (LPBF) benchmark experiments to perform a comprehensive model assessment, including verification, validation, sensitivity analysis, and uncertainty quantification. The RADICAL-EnTK workflow manager was used to perform an ensemble of heat transport, solidification, and mechanical response simulations on the exascale computer Frontier, considering uncertainties in critical model inputs such as laser spot size and nucleation parameters, and consisting of 125 explicit grain structure simulations and 7875 crystal plasticity simulations. For a selected location within the Inconel 625 AMBench-2018 test artifact, sensitivity analysis and uncertainty quantification were performed using the predicted distributions of grain structure and mechanical properties. Qualitative agreement was found between the predicted grain size and texture and the observed AMBench-2018 microstructure, the mean predicted yield stress was within 5% of the experimental measurement mean, and the mean predicted engineering stress at 5% strain was within 10% of the experimental measurement mean. The insights gained from development and validation of the ExaAM PSP modeling framework will help guide future directions for enhancing the credibility and reliability of PSP models in AM, thereby accelerating the adoption of AM technologies in various industries.

Additive manufacturing↗

DOE OSTI2024

DeepUQ: Assessing the Aleatoric Uncertainties from two Deep Learning Methods

Assessing the quality of aleatoric uncertainty estimates from uncertainty quantification (UQ) deep learning methods is important in scientific contexts, where uncertainty is physically meaningful and important to characterize and interpret exactly. We systematically compare aleatoric uncertainty measured by two UQ techniques, Deep Ensembles (DE) and Deep Evidential Regression (DER). Our method focuses on both zero-dimensional (0D) and two-dimensional (2D) data, to explore how the UQ methods function for different data dimensionalities. We investigate uncertainty injected on the input and output variables and include a method to propagate uncertainty in the case of input uncertainty so that we can compare the predicted aleatoric uncertainty to the known values. We experiment with three levels of noise. The aleatoric uncertainty predicted across all models and experiments scales with the injected noise level. However, the predicted uncertainty is miscalibrated to $\rm{std}(\sigma_{\rm al})$ with the true uncertainty for half of the DE experiments and almost all of the DER experiments. The predicted uncertainty is the least accurate for both UQ methods for the 2D input uncertainty experiment and the high-noise level. While these results do not apply to more complex data, they highlight that further research on post-facto calibration for these methods would be beneficial, particularly for high-noise and high-dimensional settings.

Nevin, Rebecca↗