SEARCH · Engineering Papers
Results for “Uncertainties”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
NOVEL APPLICATION OF UNCERTAINTY ANALYSIS TO SEVERE ACCIDENTS: INTERROGATING MODEL FORM UNCERTAINTY AND DEVELOPING INSIGHTS INTO SEVERE ACCIDENT PROGRESSION.
Abstract not provided.
Uncertainty Quantification of Moment Tensor Inversions due to Earth Model Uncertainty.
Abstract not provided.
Assessment of Radiative Heating Uncertainty for Hyperbolic Earth Entry
This paper investigates the shock-layer radiative heating uncertainty for hyperbolic Earth entry, with the main focus being a Mars return. In Part I of this work, a baseline simulation approach involving the LAURA Navier-Stokes code with coupled ablation and radiation is presented, with the HARA radiation code being used for the radiation predictions. Flight cases representative of peak-heating Mars or asteroid return are de ned and the strong influence of coupled ablation and radiation on their aerothermodynamic environments are shown. Structural uncertainties inherent in the baseline simulations are identified, with turbulence modeling, precursor absorption, grid convergence, and radiation transport uncertainties combining for a +34% and ..24% structural uncertainty on the radiative heating. A parametric uncertainty analysis, which assumes interval uncertainties, is presented. This analysis accounts for uncertainties in the radiation models as well as heat of formation uncertainties in the flow field model. Discussions and references are provided to support the uncertainty range chosen for each parameter. A parametric uncertainty of +47.3% and -28.3% is computed for the stagnation-point radiative heating for the 15 km/s Mars-return case. A breakdown of the largest individual uncertainty contributors is presented, which includes C3 Swings cross-section, photoionization edge shift, and Opacity Project atomic lines. Combining the structural and parametric uncertainty components results in a total uncertainty of +81.3% and ..52.3% for the Mars-return case. In Part II, the computational technique and uncertainty analysis presented in Part I are applied to 1960s era shock-tube and constricted-arc experimental cases. It is shown that experiments contain shock layer temperatures and radiative ux values relevant to the Mars-return cases of present interest. Comparisons between the predictions and measurements, accounting for the uncertainty in both, are made for a range of experiments. A measure of comparison quality is de ned, which consists of the percent overlap of the predicted uncertainty bar with the corresponding measurement uncertainty bar. For nearly all cases, this percent overlap is greater than zero, and for most of the higher temperature cases (T >13,000 K) it is greater than 50%. These favorable comparisons provide evidence that the baseline computational technique and uncertainty analysis presented in Part I are adequate for Mars-return simulations. In Part III, the computational technique and uncertainty analysis presented in Part I are applied to EAST shock-tube cases. These experimental cases contain wavelength dependent intensity measurements in a wavelength range that covers 60% of the radiative intensity for the 11 km/s, 5 m radius flight case studied in Part I. Comparisons between the predictions and EAST measurements are made for a range of experiments. The uncertainty analysis presented in Part I is applied to each prediction, and comparisons are made using the metrics defined in Part II. The agreement between predictions and measurements is excellent for velocities greater than 10.5 km/s. Both the wavelength dependent and wavelength integrated intensities agree within 30% for nearly all cases considered. This agreement provides confidence in the computational technique and uncertainty analysis presented in Part I, and provides further evidence that this approach is adequate for Mars-return simulations. Part IV of this paper reviews existing experimental data that include the influence of massive ablation on radiative heating. It is concluded that this existing data is not sufficient for the present uncertainty analysis. Experiments to capture the influence of massive ablation on radiation are suggested as future work, along with further studies of the radiative precursor and improvements in the radiation properties of ablation products.
Evidential Deep Learning: Enhancing Predictive Uncertainty Estimation for Earth System Science Applications
Abstract Robust quantification of predictive uncertainty is a critical addition needed for machine learning applied to weather and climate problems to improve the understanding of what is driving prediction sensitivity. Ensembles of machine learning models provide predictive uncertainty estimates in a conceptually simple way but require multiple models for training and prediction, increasing computational cost and latency. Parametric deep learning can estimate uncertainty with one model by predicting the parameters of a probability distribution but does not account for epistemic uncertainty. Evidential deep learning, a technique that extends parametric deep learning to higher-order distributions, can account for both aleatoric and epistemic uncertainties with one model. This study compares the uncertainty derived from evidential neural networks to that obtained from ensembles. Through applications of the classification of winter precipitation type and regression of surface-layer fluxes, we show evidential deep learning models attaining predictive accuracy rivaling standard methods while robustly quantifying both sources of uncertainty. We evaluate the uncertainty in terms of how well the predictions are calibrated and how well the uncertainty correlates with prediction error. Analyses of uncertainty in the context of the inputs reveal sensitivities to underlying meteorological processes, facilitating interpretation of the models. The conceptual simplicity, interpretability, and computational efficiency of evidential neural networks make them highly extensible, offering a promising approach for reliable and practical uncertainty quantification in Earth system science modeling. To encourage broader adoption of evidential deep learning, we have developed a new Python package, Machine Integration and Learning for Earth Systems (MILES) group Generalized Uncertainty for Earth System Science (GUESS) (MILES-GUESS) ( https://github.com/ai2es/miles-guess ), that enables users to train and evaluate both evidential and ensemble deep learning. Significance Statement This study demonstrates a new technique, evidential deep learning, for robust and computationally efficient uncertainty quantification in modeling the Earth system. The method integrates probabilistic principles into deep neural networks, enabling the estimation of both aleatoric uncertainty from noisy data and epistemic uncertainty from model limitations using a single model. Our analyses reveal how decomposing these uncertainties provides valuable insights into reliability, accuracy, and model shortcomings. We show that the approach can rival standard methods in classification and regression tasks within atmospheric science while offering practical advantages such as computational efficiency. With further advances, evidential networks have the potential to enhance risk assessment and decision-making across meteorology by improving uncertainty quantification, a longstanding challenge. This work establishes a strong foundation and motivation for the broader adoption of evidential learning, where properly quantifying uncertainties is critical yet lacking.
SLS Integrated Modal Test Uncertainty Quantification using the Hybrid Parametric Variation Method
Uncertainty in structural loading during launch is a significant concern in the development of spacecraft and launch vehicles. Small variations in launch vehicle and payload mode shapes and their interaction can result in significant variation in system loads. In many cases involving large aerospace systems it is difficult, not economical, or impossible to perform a system modal test. However, it is still vital to obtain test results that can be compared with analytical predictions to validate models. Instead, the “Building Block Approach” is used in which system components are tested individually. Component models are correlated and updated to agree as best they can with test results. The Space Launch System consists of a number of components that are assembled into a launch vehicle. Finite element models of the components are developed, reduced to Hurty/Craig-Bampton models and assembled to represent different phases of flight. The only opportunity to obtain modal test data from an assembled Space Launch System will be during the Integrated Modal Test. There is always uncertainty in every model, which flows into uncertainty in predicted system results. Uncertainty Quantification is used to determine statistical bounds on prediction accuracy based on model uncertainty. For the Space Launch System, model uncertainty is at the Hurty/Craig-Bampton component level. Uncertainty in the Hurty/Craig-Bampton components is quantified using the hybrid parametric variation approach that combines parametric and nonparametric uncertainty. Uncertainty in model form is one of the biggest contributors to uncertainty in complex built-up structures. This type of uncertainty cannot be represented by variations infinite element model input parameters and thus cannot be included in a parametric approach. However, model-form uncertainty can be modeled using a nonparametric approach based on random matrix theory. The hybrid parametric variation method requires the selection of dispersion values for the Hurty/Craig-Bampton fixed-interface eigenvalues, and the Hurty/Craig-Bampton stiffness matrices. Component test/analysis frequency error is used to identify the fixed-interface eigenvalue dispersions, while test/analysis cross-orthogonality is used to identify stiffness dispersion values. The hybrid parametric variation uncertainty quantification approach is applied to the Space Launch System Integrated Modal Test configuration. Monte Carlo analysis is performed, and statistics are determined for modal correlation metrics, frequency response from Integrated Modal Test shakers to selected accelerometers, as well as other metrics for determining how well target modes are excited and identified. If the predicted uncertainty envelopes future Integrated Modal Test results, then there will be increased confidence in the utility of the component-based hybrid parametric variation uncertainty quantification approach.
Meta-uncertainty for particle image velocimetry
Uncertainty quantification for Particle Image Velocimetry (PIV) is critical for comparing experimentally measured flow fields with Computational Fluid Dynamics (CFD) results, and model design and validation. However, PIV features a complex measurement chain with coupled, non-linear error sources, and quantifying the uncertainty is challenging. Multiple assessments show that none of the current methods can reliably measure the actual uncertainty across a wide range of experiments, and estimates can vary. Because the current methods differ in assumptions regarding the measurement process and calculation procedures, it is not clear which method is best to use for an experiment where the error distribution is unknown. To address this issue, we propose a method to estimate an uncertainty method's sensitivity and reliability, termed the Meta-Uncertainty. The novel approach is automated, local, and instantaneous, and based on perturbation of the recorded particle images. We developed an image perturbation scheme based on adding random unmatched particles to the interrogation window pair considering the signal-to-noise (SNR) of the correlation plane. Each uncertainty scheme's response to several trials of random particle addition is used to estimate a reliability metric, defined as the rate of change of the inter-quartile range (IQR) of the uncertainties with increasing levels of particle addition. We also propose applying the meta-uncertainty as a weighting metric to combine uncertainty estimates from individual schemes, based on ideas from the consensus forecasting literature. We use planar and stereo PIV measurements across a range of canonical flows to assess the performance of the uncertainty schemes. Further, a novel method is introduced to assess an uncertainty scheme's performance based on a quantile comparison of the error and uncertainty distributions, generalizing the current method of comparing the RMS of the two distributions. Here, the results show that the combined uncertainty method outperforms the individual methods, and this work establishes the meta-uncertainty as a useful reliability assessment tool for PIV uncertainty quantification.
Uncertainty Quantification of Calculated Temperatures for the AGR 5/6/7 Experiment
This report documents the quantification of uncertainty of the calculated temperature data for the Advanced Gas Reactor (AGR) 5/6/7 fuel irradiation experiment conducted in the Advanced Test Reactor at Idaho National Laboratory in support of the Advanced Reactor Technologies? research and development program. Recognizing uncertainties inherent in physics and thermal simulations of the AGR 5/6/7 capsules, the results of the numerical simulations are used in combination with statistical analysis methods to improve qualification of measured data. The calculated fuel temperatures for AGR tests are also used for validation of the fission product transport and fuel performance simulation models. These crucial roles of the calculated fuel temperatures in ensuring achievement of the AGR experimental program objectives require accurate determination of the model temperature uncertainties. This report covers temperature uncertainty results for each of the five AGR 5/6/7 capsules. To quantify the uncertainty of calculated temperatures determined using the ABAQUS finite element heat transfer code, this study identifies and analyzes model parameters of potential importance to the calculated temperatures of fuel compacts and thermocouples. The selection of input parameters for uncertainty quantification is based on the ranking of their influences upon temperature predictions. Thus, selected input parameters include those with high sensitivity and those with the largest uncertainty. Propagation of model parameter uncertainty and sensitivity is then used to quantify the overall uncertainty of calculated temperatures. Measurement uncertainty, analysis of modeling assumptions, and expert judgment are used as the basis to quantify the uncertainty range for selected input parameters. The input uncertainties are dynamic, accounting for the effect of unplanned events and changes in thermal properties of capsule components over extended exposure to high temperatures and fast neutron irradiation. The sensitivity analysis performed in this work went beyond the traditional local sensitivity. Using experimental design, analysis of pairwise interactions of model parameters was performed to establish sufficiency of the time dependent first order (linear) expansion terms in constructing the temperature response surface. To achieve completeness, uncertainty propagation made use of pairwise noise correlations of model parameters. Furthermore, using an interpolation scheme over the input parameter domain, the analysis obtains time dependent sensitivity over the test campaign duration. This allows computation of uncertainty for the calculated fuel temperatures and the calculated graphite temperatures at thermocouple locations during the entire irradiation period.
Uncertainty Modeling for Structural Control Analysis and Synthesis
The development of an accurate model of uncertainties for the control of structures that undergo a change in operational environment, based solely on modeling and experimentation in the original environment is studied. The application used throughout this work is the development of an on-orbit uncertainty model based on ground modeling and experimentation. A ground based uncertainty model consisting of mean errors and bounds on critical structural parameters is developed. The uncertainty model is created using multiple data sets to observe all relevant uncertainties in the system. The Discrete Extended Kalman Filter is used as an identification/parameter estimation method for each data set, in addition to providing a covariance matrix which aids in the development of the uncertainty model. Once ground based modal uncertainties have been developed, they are localized to specific degrees of freedom in the form of mass and stiffness uncertainties. Two techniques are presented: a matrix method which develops the mass and stiffness uncertainties in a mathematical manner; and a sensitivity method which assumes a form for the mass and stiffness uncertainties in macroelements and scaling factors. This form allows the derivation of mass and stiffness uncertainties in a more physical manner. The mass and stiffness uncertainties of the ground based system are then mapped onto the on-orbit system, and projected to create an analogous on-orbit uncertainty model in the form of mean errors and bounds on critical parameters. The Middeck Active Control Experiment is introduced as experimental verification for the localization and projection methods developed. In addition, closed loop results from on-orbit operations of the experiment verify the use of the uncertainty model for control analysis and synthesis in space.
Quantifying the Uncertainty of the Future Hydrological Impacts of Climate Change: Comparative Analysis of an Advanced Hierarchical Sensitivity in Humid and Semiarid Basins
Comparison and quantification of different uncertainties of future climate change involved in the modeling of a hydrological system are highly important for both hydrological modelers and policy-makers. However, few studies have accurately estimated the relative importance of different sources of uncertainty at different spatiotemporal scales. Here, a hierarchical sensitivity analysis framework (HSAF) incorporated with a variance-based global sensitivity analysis is developed to quantify the spatiotemporal contributions of different uncertainties in hydrological impacts of climate change in two different climatic (humid and semiarid) basins in China. The uncertainty sources include three emission scenarios (ESs), 20 global climate models (GCs), three hydrological models (HMs), and the associated sensitive hydrological parameters (PAs) screened and sampled by the Morris and Latin hypercube sampling methods, respectively. Further, the results indicate that the overall trend of uncertainty is PA > HM > GC > ES, but their uncertainties have discrepancies in projections of different hydrological variables. The HM uncertainty in annual and monthly discharge projections is generally larger than the PA uncertainty in the humid basin than semiarid basin. The PA has greater uncertainty in extreme hydrological event (annual peak discharge) projections than in annual discharge projections for both basins (particularly for the humid basin), but contributes larger uncertainty to annual and monthly discharge projections in the semiarid basin than humid basin. The GC contributes larger uncertainty in all the hydrological variables projections in the humid basin than semiarid basin, while the ES uncertainty is rather limited in both basins. Overall, our results suggest there is greater spatiotemporal variability of hydrological uncertainty in more arid regions.
Measurement Uncertainty in One-Of-A-Kind Event Data Analysis
A golden standard in science is to repeat an experiment a statistically significant number of times, recording data using the same set of detectors and the same data analysis methodology. In such case experimental error includes both the range of true values generated by repetitions of the experiment, and measurement uncertainty caused by the detector. They are independent. It is a huge and too frequently used simplification, to assume that one can measure multiple repetitions of an identical experiment, resulting in identical true experimental value. Repetitions, as similar is it is experimentally achievable, have unavoidable built-in differences resulting in a range of the true values rather than in a single value. When modern, very sensitive and well calibrated measurement systems are used, this range is not negligible, and sometimes dominates over the measurement uncertainty. Range of true values depends on built-in differences in physics of the experiment. Stochastic physical processes result typically in a broader range of true values than non-stochastic processes do. Measurement uncertainty depends on a measurement method (properties of the detector not of the experiment). Modern measurement methods, including digital ones, frequently make the measurement uncertainty very small. When data from one–of –a kind experiment are analyzed, only the measurement uncertainty is reported. It provides no information about the range of true experimental values, neither about reliability of a reported data point. Reliability of a data point is in general independent from its measurement uncertainty. However, in practice reliable measurement methods frequently have high measurement uncertainty, while low reliability methods are applied to limit measurement uncertainty. Comparison of reliable data with high measurement uncertainty to not so reliable data measured with low uncertainty is discussed – in different scenarios different data analysis methods are applicable. Methods for data analysis from an experiment repeated statistically significant number of times are very well developed. They do not require a detailed expertise in physics of an experiment, nor in the properties of the measurement system used, and meaning of the reported uncertainty is well understood in any scientific community. It all changes when data from one-of-a-kind experiment is analyzed. Analyst’s expertise is required both in the physics of the experiment and in all aspects of the measurement system, all possible malfunctions. Data users must remember that only measurement uncertainty is reported from any one-of-a-kind experiment. Theory with simulations may provide estimation of expected built-in differences in the experiment, and by this of expected range of true values for a given experiment; yet measurement uncertainty can never be used in place of the range of true experimental values.
E-Area Low Level Waste Generator Inventory Uncertainty Estimation
This report describes the method used to estimate the uncertainty and bias in the low-level waste inventory, as reported by Waste Generators, for waste disposed in the E-Area Low-Level Waste Facility at SRS. The waste cut uncertainty and bias is summed to calculate the current total uncertainty and bias in each disposal unit in the E-Area Low-Level Waste Facility. This information is used to estimate the future uncertainty and bias when all disposal units are closed. For most waste cuts, there are two areas of inventory uncertainty and bias: in the isotopic characterization of the waste stream and in the measurement of total waste cut activity. Waste stream characterization is accomplished through a combination of process knowledge and analytical measurements. This characterization is documented by a technical baseline that describes the relative quantity of each isotope present in the waste stream. The total activity of each individual waste cut is usually measured directly or is calculated through a ratio of activity per weight or activity per volume. The waste cut total activity is multiplied by the waste stream isotopic fraction to calculate the total isotopic activity present in the waste cut. The waste cut uncertainty and bias is calculated by analyzing the uncertainties and biases in the waste stream characterization and waste cut measurements and adding the uncertainties in quadrature and multiplying the reported isotopic activities by the bias factors. The current isotopic inventory, uncertainty, and bias of a disposal unit is the sum of the currently disposed individual waste cut isotopic activities, uncertainties, and biases. The future uncertainty and bias is estimated based on the average “worst-case” (largest) uncertainties and biases of currently closed and operational disposal units. This report details the method, data sources, and calculation techniques used to calculate these values. It also presents some of the difficulties in working with the reported data and gives examples of the calculations and analysis. The software used to carryout the analysis is included and the location of the complete individual analyses is given. After discussion with waste generators, examination of multiple waste characterization documents, and a review of SRS Manual 1S, the reported ELLWF radionuclide activities are clearly conservative and likely represent reasonable upper bounds for the true activities. However, quantifying the bias and uncertainty in each waste cut requires detailed analysis of the documentation for each waste cut measurement and waste stream characterization.
Estimating pixel-level uncertainty in ocean color retrievals from MODIS
The spectral distribution of marine remote sensing reflectance, R(rs), is the fundamental measurement of ocean color science, from which a host of bio-optical and biogeochemical properties of the water column can be derived. Estimation of uncertainty in these derived properties is thus dependent on knowledge of the uncertainty in satellite-retrieved R(rs) (u(c)(R(rs))) at each pixel. Uncertainty in R(rs), in turn, is dependent on the propagation of various uncertainty sources through the R(rs) retrieval process, namely the atmospheric correction (AC). A derivative-based method for uncertainty propagation is established here to calculate the pixel-level uncertainty in R(rs), as retrieved using NASA’s multiple-scattering epsilon (MSEPS) AC algorithm and verified using Monte Carlo (MC) analysis. The approach is then applied to measurements from the Moderate Resolution Imaging Spectroradiometer (MODIS) on the Aqua satellite, with uncertainty sources including instrument random noise, instrument systematic uncertainty, and forward model uncertainty. The uc(Rrs) is verified by comparison with statistical analysis of coincident retrievals from MODIS and in situ Rrs measurements, and our approach performs well in most cases. Based on analysis of an example 8-day global products, we also show that relative uncertainty in R(rs) at blue bands has a similar spatial pattern to the derived concentration of the phytoplankton pigment chlorophyll-a (chl-a), and around 7.3%, 17.0%, and 35.2% of all clear water pixels (chl-a ≤ 0.1 mg/cu.m) with valid u(c)(R(rs)) have a relative uncertainty ≤ 5% at bands 412 nm, 443 nm, and 488 nm respectively, which is a common goal of ocean color retrievals for clear waters. While the analysis shows that u(c)(R(rs)) calculated from our derivative-based method is reasonable, some issues need further investigation, including improved knowledge of forward model uncertainty and systematic uncertainty in instrument calibration.
Uncertainty Analysis of the NASA Glenn 8- by 6-Foot Supersonic Wind Tunnel 2019 Characterization Test 14-Foot, 5.8 Percent Test Section Porosity
This paper presents methods and results of a measurement uncertainty analysis that was performed for the 8- by 6-foot Supersonic Wind Tunnel located at the NASA Glenn Research Center. The uncertainty analysis is based on data collected during a characterization test following major structural modifications to the facility and upgrades to the facility’s data and control systems. The statistical methods and engineering judgments used to estimate elemental uncertainties are described in this report. The Monte Carlo method of propagating uncertainty was selected to determine the uncertainty of calculated variables of interest. A description of the Monte Carlo method as applied for this analysis is provided. The primary variable of interest (VOI) for this facility is free stream Mach number. In addition to determining the uncertainty in Mach number, the uncertainty in free stream values of static pressure, total pressure, dynamic pressure, total temperature, static temperature, Reynolds number, and air speed were also calculated. Uncertainty results are presented as random (unpredictable variation in repeated measurements), systematic (potential offset between observed and true value), and total (random and systematic combined) uncertainty for Mach number. Systematic uncertainty results are presented for the other VOIs. Individual uncertainty sources are presented both dimensionally and as percent contributions to uncertainty in all VOIs, to aid in the identification of primary uncertainty sources.
Characterizing Epistemic Uncertainty for Launch Vehicle Designs
NASA Probabilistic Risk Assessment (PRA) has the task of estimating the aleatory (randomness) and epistemic (lack of knowledge) uncertainty of launch vehicle loss of mission and crew risk, and communicating the results. Launch vehicles are complex engineered systems designed with sophisticated subsystems that are built to work together to accomplish mission success. Some of these systems or subsystems are in the form of heritage equipment, while some have never been previously launched. For these cases, characterizing the epistemic uncertainty is of foremost importance, and it is anticipated that the epistemic uncertainty of a modified launch vehicle design versus a design of well understood heritage equipment would be greater. For reasons that will be discussed, standard uncertainty propagation methods using Monte Carlo simulation produce counter intuitive results, and significantly underestimate epistemic uncertainty for launch vehicle models. Furthermore, standard PRA methods, such as Uncertainty-Importance analyses used to identify components that are significant contributors to uncertainty, are rendered obsolete, since sensitivity to uncertainty changes are not reflected in propagation of uncertainty using Monte Carlo methods. This paper provides a basis of the uncertainty underestimation for complex systems and especially, due to nuances of launch vehicle logic, for launch vehicles. It then suggests several alternative methods for estimating uncertainty and provides examples of estimation results. Lastly, the paper describes how to implement an Uncertainty-Importance analysis using one alternative approach, describes the results, and suggests ways to reduce epistemic uncertainty by focusing on additional data or testing of selected components.
Characterizing Epistemic Uncertainty for Launch Vehicle Designs
NASA Probabilistic Risk Assessment (PRA) has the task of estimating the aleatory (randomness) and epistemic (lack of knowledge) uncertainty of launch vehicle loss of mission and crew risk and communicating the results. Launch vehicles are complex engineered systems designed with sophisticated subsystems that are built to work together to accomplish mission success. Some of these systems or subsystems are in the form of heritage equipment, while some have never been previously launched. For these cases, characterizing the epistemic uncertainty is of foremost importance, and it is anticipated that the epistemic uncertainty of a modified launch vehicle design versus a design of well understood heritage equipment would be greater. For reasons that will be discussed, standard uncertainty propagation methods using Monte Carlo simulation produce counter intuitive results and significantly underestimate epistemic uncertainty for launch vehicle models. Furthermore, standard PRA methods such as Uncertainty-Importance analyses used to identify components that are significant contributors to uncertainty are rendered obsolete since sensitivity to uncertainty changes are not reflected in propagation of uncertainty using Monte Carlo methods.This paper provides a basis of the uncertainty underestimation for complex systems and especially, due to nuances of launch vehicle logic, for launch vehicles. It then suggests several alternative methods for estimating uncertainty and provides examples of estimation results. Lastly, the paper shows how to implement an Uncertainty-Importance analysis using one alternative approach, describes the results, and suggests ways to reduce epistemic uncertainty by focusing on additional data or testing of selected components.
DeepUQ: Assessing the Aleatoric Uncertainties from two Deep Learning Methods
Assessing the quality of aleatoric uncertainty estimates from uncertainty quantification (UQ) deep learning methods is important in scientific contexts, where uncertainty is physically meaningful and important to characterize and interpret exactly. We systematically compare aleatoric uncertainty measured by two UQ techniques, Deep Ensembles (DE) and Deep Evidential Regression (DER). Our method focuses on both zero-dimensional (0D) and two-dimensional (2D) data, to explore how the UQ methods function for different data dimensionalities. We investigate uncertainty injected on the input and output variables and include a method to propagate uncertainty in the case of input uncertainty so that we can compare the predicted aleatoric uncertainty to the known values. We experiment with three levels of noise. The aleatoric uncertainty predicted across all models and experiments scales with the injected noise level. However, the predicted uncertainty is miscalibrated to $\rm{std}(\sigma_{\rm al})$ with the true uncertainty for half of the DE experiments and almost all of the DER experiments. The predicted uncertainty is the least accurate for both UQ methods for the 2D input uncertainty experiment and the high-noise level. While these results do not apply to more complex data, they highlight that further research on post-facto calibration for these methods would be beneficial, particularly for high-noise and high-dimensional settings.
Uncertainty Analysis of the CE-22 Advanced Nozzle Test Facility
This paper presents methods and results of a detailed measurement uncertainty analysis that was performed for the Advanced Nozzle Test Facility, CE-22, located at the NASA Glenn Research Center. Results for the uncertainty in thrust and flow coefficients, in addition to other variables of interest, are provided. Results are presented separately as random uncertainty (characterizing errors inherent to instrument or measurement environment, impacting repeatability), systematic uncertainty (capturing inaccuracies due to measurement process, calibration, installation effects or other sources which may introduce bias), and total combined uncertainty. The statistical methods and engineering judgments used to estimate elemental uncertainties are described. The analysis tool MANTUS (Measurement Analysis Tool for Uncertainty in Systems) was used to quantify instrumentation system uncertainty, and statistical analysis and engineering judgment were used to quantify other random and systematic uncertainty sources. The Monte Carlo method was used to propagate systematic and random elemental uncertainties to determine the uncertainties of various calculated variables of interest.