Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Bayesian Neural Network”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13

Bayesian averaging for ground state masses of atomic nuclei in a Machine Learning approach

We present global predictions of the ground state mass of atomic nuclei based on a novel Machine Learning algorithm. We combine precision nuclear experimental measurements together with theoretical predictions of unmeasured nuclei. This hybrid data set is used to train a probabilistic neural network. In addition to training on this data, a physics-based loss function is employed to help refine the solutions. The resultant Bayesian averaged predictions have excellent performance compared to the testing set and come with well-quantified uncertainties which are critical for contemporary scientific applications. We assess extrapolations of the model’s predictions and estimate the growth of uncertainties in the region far from measurements.

74 ATOMIC AND MOLECULAR PHYSICS↗

nautilus : boosting Bayesian importance nested sampling with deep learning

ABSTRACT We introduce a novel approach to boost the efficiency of the importance nested sampling (INS) technique for Bayesian posterior and evidence estimation using deep learning. Unlike rejection-based sampling methods such as vanilla nested sampling (NS) or Markov chain Monte Carlo (MCMC) algorithms, importance sampling techniques can use all likelihood evaluations for posterior and evidence estimation. However, for efficient importance sampling, one needs proposal distributions that closely mimic the posterior distributions. We show how to combine INS with deep learning via neural network regression to accomplish this task. We also introduce nautilus, a reference open-source python implementation of this technique for Bayesian posterior and evidence estimation. We compare nautilus against popular NS and MCMC packages, including emcee, dynesty, ultranest, and pocomc, on a variety of challenging synthetic problems and real-world applications in exoplanet detection, galaxy SED fitting and cosmology. In all applications, the sampling efficiency of nautilus is substantially higher than that of all other samplers, often by more than an order of magnitude. Simultaneously, nautilus delivers highly accurate results and needs fewer likelihood evaluations than all other samplers tested. We also show that nautilus has good scaling with the dimensionality of the likelihood and is easily parallelizable to many CPUs.

97 MATHEMATICS AND COMPUTING↗

Machine-learning-based inversion of nuclear responses

A microscopic description of the interaction of atomic nuclei with external electroweak probes is required for elucidating aspects of short-range nuclear dynamics and for the correct interpretation of neutrino oscillation experiments. Nuclear quantum Monte Carlo methods infer the nuclear electroweak response functions from their Laplace transforms. Inverting the Laplace transform is a notoriously ill-posed problem; and Bayesian techniques, such as maximum entropy, are typically used to reconstruct the original response functions in the quasielastic region. In this work, we present a physics-informed artificial neural network architecture suitable for approximating the inverse of the Laplace transform. Utilizing simulated, albeit realistic, electromagnetic response functions, we show that this physics-informed artificial neural network outperforms maximum entropy in both the low-energy transfer and the quasielastic regions, thereby allowing for robust calculations of electron scattering and neutrino scattering on nuclei and inclusive muon capture rates.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Stopping criteria for ending autonomous, single detector radiological source searches

While the localization of radiological sources has traditionally been handled with statistical algorithms, such a task can be augmented with advanced machine learning methodologies. The combination of deep and reinforcement learning has provided learning-based navigation to autonomous, single-detector, mobile systems. However, these approaches lacked the capacity to terminate a surveying/search task without outside influence of an operator or perfect knowledge of source location (defeating the purpose of such a system). Two stopping criteria are investigated in this work for a machine learning navigated system: one based upon Bayesian and maximum likelihood estimation (MLE) strategies commonly used in source localization, and a second providing the navigational machine learning network with a “stop search” action. A convolutional neural network was trained via reinforcement learning in a 10 m × 10 m simulated environment to navigate a randomly placed detector-agent to a randomly placed source of varied strength (stopping with perfect knowledge during training). The network agent could move in one of four directions (up, down, left, right) after taking a 1 s count measurement at the current location. During testing, the stopping criteria for this navigational algorithm was based upon a Bayesian likelihood estimation technique of source presence, updating this likelihood after each step, and terminating once the confidence of the source being in a single location exceeded 0.9. A second network was trained and tested with similar architecture as the previous but which contained a fifth action: for self-stopping. The accuracy and speed of localization with set detector and source initializations were compared over 50 trials of MLE-Bayesian approach and 1000 trials of the CNN with self-stopping. The statistical stopping condition yielded a median localization error of ~1.41 m and median localization speed of 12 steps. The machine learning stopping condition yielded a median localization error of 0 m and median localization speed of 17 steps. This work demonstrated two stopping criteria available to a machine learning guided, source localization system.

38 RADIATION CHEMISTRY, RADIOCHEMISTRY, AND NUCLEA↗

Empirical radius formulas for canonical neutron stars from bidirectionally selecting features of equations of state in extended Bayesian analyses of observational data

Significant advancement in Bayesian inference of nuclear equation of state (EOS) from gravitational wave and x-ray observations of neutron stars (NSs) has been made by the nuclear astrophysics community especially since GW170817. By extending the traditional Bayesian analysis which normally ends at presenting the marginalized posterior probability distribution functions (PDFs) of individual EOS parameters and their correlations (or sometimes only the Pearson correlation coefficients which are only reliably useful when the variables are linearly correlated while they are actually often not), we search for a data-driven and robust empirical formula for the radius 𝑅 1.4 of canonical NSs in terms of the characteristic EOS parameters (features). We also identify the single most important but currently poorly known EOS parameter for determining the 𝑅 1.4 . Using three regression-model-building methodologies: bidirectional stepwise feature selection, least absolute shrinkage selection operator (LASSO) regression, and neural network regression on a large set of posterior EOSs and the corresponding 𝑅 1.4 values inferred from earlier comprehensive Bayesian analyses of NS observational data, we systematically and rigorously develop the most probable 𝑅 1.4 formulas with varying statistical accuracy and technical complexity. Here, the most important EOS parameters for determining 𝑅 1.4 are found consistently in each of the feature selection processes to be (in order of decreasing importance): curvature 𝐾 sym , slope 𝐿, skewness 𝐽 sym of nuclear symmetry energy, skewness 𝐽 0 , incompressibility 𝐾 0 of symmetric nuclear matter, and the magnitude 𝐸 sym ⁡(𝜌 0 ) of symmetry energy at the saturation density 𝜌 0 of nuclear matter.

Bayesian methods↗

MOOSE ProbML: Parallelizable Probabilistic Machine Learning and Uncertainty Quantification Capabilities

The Multiphysics Object Oriented Simulation Environment (MOOSE) is a widely used open- source finite element software for performing multiphysics multiscale simulations in a massively parallel fashion. Recently, the computational team at Idaho National Laboratory (INL) has implemented Probabilistic Machine Learning (ProbML) capabilities in MOOSE—in a parallelized fashion—and enable active learning with large-scale computational models for tasks such as surrogate model development, scale bridging, forward/inverse uncertainty quantification (UQ), Bayesian optimization, etc. This presentation summarizes these developments in MOOSE along with demonstrations on several real applications relevant to nuclear energy. At the fundamental level, samplers like Monte Carlo/Latin Hypercube, variance reduction, parallelized Markov Chain Monte Carlo (MCMC) support uncertainty propagation in both forward and inverse settings. These samplers can be integrated with the Gaussian processes (GP) suite in MOOSE, which offer several variants like scalar GPs, multi-output GPs, and deep GPs, to enable active learning. These GPs can be tuned using gradient-based optimization methods like Adam and its variants or gradient-free methods like the elliptical slice sampler (a variant of MCMC adept under Gaussian settings) for more complex covariance kernels or likelihoods whose gradient computations can be cumbersome. A variety of batch acquisition functions permit parallelized evaluation of the computational model and support different learning objectives with high efficiency like Bayesian inference, global surrogate development, optimization, etc. Furthermore, libtorch integration supports training, evaluation, and re-training of neural networks and other complex machine learning models in active learning settings. The impacts of these developments are shown on several real applications: (1) nuclear fuel inverse UQ and model inadequacy assessment using the Kennedy O’Hagan framework; (2) uncertainty aware surrogate modeling for additive manufacturing to predict field quantities; (3) nuclear reactor rare events analysis; and (4) complex fluid flow prediction using a global surrogate with quantified prediction uncertainty. Finally, the outlook of MOOSE ProbML is discussed for both outer-loop and inner-loop computations in the broad view to accelerate fuels and materials qualification, address gaps in knowledge and data, and assess new reactor/fuel systems.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Lithium-Ion Battery Diagnostics Using Electrochemical Impedance via Machine-Learning

Diagnosing battery states such as health, state-of-charge, or temperature is crucial for ensuring the safety and reliability of electrochemical energy storage systems. While some states, such as temperature, may be measured using cheap sensors, accurate diagnosis of battery health metrics usually requires time-consuming performance measurements, making them infeasible for use in real-world operation. These health metrics can be measured during lab-testing and then estimated on-line using predictive life models or via state observer algorithms such as Kalman filters, but these predictive methods should be supplemented by actual measurement of battery health whenever possible to ensure reliability. Rapid measurement of battery health may be done by various types of fast diagnostic techniques such as electrochemical impedance spectroscopy (EIS), which can be performed in only a few minutes and require only a fraction of the energy and power needed for a full charge and discharge measurement. But there is a substantial challenge for estimating battery health using EIS data, as EIS is sensitive to cell temperature, state-of-charge, current, and resting time in addition to health. Thus, utilizing EIS data to predict battery capacity requires correcting for all these additional variables, a task that is extremely difficult to handle analytically. This talk utilizes machine-learning methods to estimate the effectiveness of battery capacity prediction from EIS data, leveraging a data set of hundreds of EIS measurements recorded at varying temperature and state-of-charge throughout a 500-day aging study of 32 commercial, large-format NMC-Graphite lithium-ion batteries. Using EIS as input to machine-learning models is complicated by the nonlinear response of impedance to battery health, temperature, and state-of-charge, as well as the collinearity between the impedance response at neighboring frequencies, which can easily lead to overfit models. To train robust models, features from EIS data need to be extracted from the data or some subset of critical frequencies selected. Many approaches for extracting and selecting features from EIS data from electrochemical analysis and machine-learning fields were identified for analysis: using the entire raw spectra; selection of one, two, or many frequencies from the entire spectra; selecting interesting points from the EIS measurement using domain knowledge; fitting EIS with an equivalent-circuit model; calculating statistics on the raw impedance values; and reducing the dimensionality of the data using unsupervised linear (principal component analysis) and non-linear (uniform manifold approximation and projection) methods. These approaches were rigorously compared using a machine-learning pipeline approach, training linear, Gaussian process, and random forest regression models and quantifying performance using cross-validation as well as a held-out test set. An artificial neural network model trained on the raw spectra was also tested. Promising pipelines were fine-tuned via Bayesian hyperparameter optimization using cross-validation loss and training with class-specific weights to counter data set imbalance. The most reliable method for utilizing impedance in this work was the selection of two optimal frequencies through an exhaustive search, resulting in about 2% mean absolute error on test data for both Gaussian process and random forest model architectures. Interrogation of a variety of models reveals critical frequencies of 100 Hz and 103 Hz for this data set, though the optimal set of frequencies is not necessarily intuitive, i.e., the best performing models are not simply those that use impedance at frequencies that have the highest correlation to the relative discharge capacity. The best performing model is an ensemble model, which is able to predict battery capacity with 1.9% mean absolute error for unseen cells using impedance recorded at a variety of temperatures and states-of-charge.

battery↗

Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment

Training neural networks for neuromorphic deployment is non-trivial. There have been a variety of approaches proposed to adapt back-propagation or back-propagation-like algorithms appropriate for training. Considering that these networks often have very different performance characteristics than traditional neural networks, it is often unclear how to set either the network topology or the hyperparameters to achieve optimal performance. In this work, we introduce a Bayesian approach for optimizing the hyperparameters of an algorithm for training binary communication networks that can be deployed to neuromorphic hardware. We show that by optimizing the hyperparameters on this algorithm for each dataset, we can achieve improvements in accuracy over the previous state-of-the-art for this algorithm on each dataset (by up to 15 percent). This jump in performance continues to emphasize the potential when converting traditional neural networks to binary communication applicable to neuromorphic hardware.

Parsa, Maryam↗

Robustness of Deep Learning Classification to Adversarial Input on GPUs: Asynchronous Parallel Accumulation Is a Source of Vulnerability

The ability of machine learning (ML) classification models to resist small, targeted input perturbations—known as adversarial attacks—is a key measure of their safety and reliability. We show that floating-point non associativity (FPNA) coupled with asynchronous parallel programming on GPUs is sufficient to result in misclassification, without any perturbation to the input. Additionally, we show that this misclassification is particularly significant for inputs close to the decision boundary and that standard adversarial robustness results may be overestimated up to 4.6 when not considering machine-level details. We first study a linear classifier, before focusing on standard Graph Neural Network (GNN) architectures and datasets used in robustness assessments. We develop a novel black-box attack using Bayesian optimization to discover external workloads that can change the instruction scheduling which bias the output of reductions on GPUs and reliably lead to misclassification. Motivated by these results, we present a new learnable permutation (LP) gradient-based approach to learning floating-point operation orderings that lead to misclassifications. The LP approach provides a worst-case estimate in a computationally efficient manner, avoiding the need to run identical experiments tens of thousands of times over a potentially large set of possible GPU states or architectures. Finally, using instrumentation-based testing, we investigate parallel reduction ordering across different GPU architectures under external background workloads, when utilizing multi-GPU virtualization, and when applying power capping. Our results demonstrate that parallel reduction ordering varies significantly across architectures under the first two conditions, substantially increasing the search space required to fully test the effects of this parallel scheduler-based vulnerability. These results and the methods developed here can help to include machine-level considerations into adversarial robustness assessments, which can make a difference in safety and mission critical applications.

Shanmugavelu, Sanjif [Maxeler Technologies, a Groq↗

A Guide to the Literature on Learning Graphical Models

This literature review discusses different methods under the general rubric of learning Bayesian networks from data, and more generally, learning probabilistic graphical models. Because many problems in artificial intelligence, statistics and neural networks can be represented as a probabilistic graphical model, this area provides a unifying perspective on learning. This paper organizes the research in this area along methodological lines of increasing complexity.

Buntine, Wray L.↗

Applying Machine Learning and Bayesian Inference to Identify and Locate Moving Anthropogenic Sources Using Distributed Acoustic Sensing Data

Distributed acoustic sensing (DAS) systems, which use existing telecommunication fibers, offer high‐resolution capabilities ideal for recording anthropogenic sources. However, the complexity of urban environments and the large amount of data recorded by DAS require automated methods to efficiently detect and categorize anthropogenic sources. Here, we evaluate how well three machine learning models (k‐nearest neighbor [k‐NN], convolutional neural networks, and recurrent‐convolutional neural networks) can identify various anthropogenic sources recorded by DAS. Our findings reveal that both k‐NN and neural network methods perform well in high signal‐to‐noise ratio (SNR) settings. However, their accuracy decreases at SNRs <4. We also use Kalman filtering, a form of Bayesian inference, on backprojected locations of these sources to recover locations that generally fall within standard smartphone Global Positioning System errors. By combining machine learning and Kalman filter results, we calculate a multidimensional model of moving anthropogenic sources. These results demonstrate the potential of DAS data in urban seismology for accurately identifying and locating such sources. Depending on the research objectives, these sources can be further studied or filtered out to improve the quality of seismic data for earthquake studies. Such methods provide a valuable tool for urban seismology and seismic hazard analysis.

Luckie, Thomas William [Sandia National Laboratori↗

SPIKE-Dx : A Low-Power High-Throughput Fault Diagnostics Tool using Spiking Neural Networks for Constrained Systems

Diagnostic systems are important for many aerospace systems, which are severely limited in available power, like cubesats or UAVs. Therefore, traditional diagnostics systems cannot be used due to their substantial footprint and constraints. In this paper, we present our very low power diagnostic tool SPIKE-DX to monitor critical systems with constrained computational and energy resources. This is made possible through spiking neural networks (SNNs), which are executable within optimized simulation environments and further implemented on on cutting-edge neuromorphic hardware. Based upon FMEA (Failure Mode and Effect Analysis) framework, Diagnostic Bayesian Networks (DBNs) can be constructed that provide powerful means for diagnostic reasoning. In this paper, we describe such DBNs and a method to automatically translate the DBN into highly structured networks of spiking neurons for execution in SPIKE-DX.

Spiking Neural Networks↗

Application of machine learning and artificial intelligence to extend EFIT equilibrium reconstruction

Recent progress in the application of machine learning (ML)/artificial intelligence (AI) algorithms to improve the Equilibrium Fitting (EFIT) code equilibrium reconstruction for fusion data analysis applications is presented. A device-independent portable core equilibrium solver capable of computing or reconstructing equilibrium for different tokamaks has been created to facilitate adaptation of ML/AI algorithms. A large EFIT database comprising of DIII-D magnetic, motional Stark effect, and kinetic reconstruction data has been generated for developments of EFIT model-order-reduction (MOR) surrogate models to reconstruct approximate equilibrium solutions. Furthermore, a neural-network MOR surrogate model has been successfully trained and tested using the magnetically reconstructed datasets with encouraging results. Other progress includes developments of a Gaussian process Bayesian framework that can adapt its many hyperparameters to improve processing of experimental input data and a 3D perturbed equilibrium database from toroidal full magnetohydrodynamic linear response modeling using the Magnetohydrodynamic Resistive Spectrum - Feedback (MARS-F) code for developments of 3D-MOR surrogate models.

Gaussian process↗

Active operator learning with predictive uncertainty quantification for partial differential equations

With the increased prevalence of neural operators being used to provide rapid solutions to partial differential equations (PDEs), understanding the accuracy of model predictions and the associated error levels is necessary for deploying reliable surrogate models in scientific applications. Existing uncertainty quantification (UQ) frameworks employ ensembles or Bayesian methods, which can incur substantial computational costs during both training and inference. Here, we propose a lightweight predictive UQ method tailored for Deep operator networks (DeepONets) that also generalizes to other operator networks. Numerical experiments on linear and nonlinear PDEs demonstrate that the framework’s uncertainty estimates are unbiased and provide accurate out-of-distribution uncertainty predictions with a sufficiently large training dataset. Our framework provides fast inference and uncertainty estimates that can efficiently drive outer-loop analyses that would be prohibitively expensive with conventional solvers. We demonstrate how predictive uncertainties can be used in the context of Bayesian optimization and active learning problems to yield improvements in accuracy and data-efficiency for outer-loop optimization procedures. In the active learning setup, we extend the framework to Fourier Neural Operators (FNO) and describe a generalized method for other operator networks. To enable real-time deployment, we introduce an inference strategy based on precomputed trunk outputs and a sparse placement matrix, reducing evaluation time by more than a factor of five. Our method provides a practical route to uncertainty-aware operator learning in time-sensitive settings.

97 MATHEMATICS AND COMPUTING↗

Radiation Source Localization Using Surrogate Models Constructed from 3-D Monte Carlo Transport Physics Simulations

Recent research has focused on the development of surrogate models for radiation source localization in a simulated urban domain. We employ the Monte Carlo N-Particle (MCNP) code to provide high- delity simulations of radiation transport within an urban domain. The model is constructed to employ a source location (x, y, z) as input and return the estimated count rate for a set of speci ed detector locations. Because MCNP simulations are computationally expensive, we develop e cient and accurate surrogate models of the detector responses. We construct surrogate models using Gaussian processes (GP) and neural networks (NN) that we train and verify using the MCNP simulations. The trained surrogate models provide an e cient framework for Bayesian inference and experimental design. We employ Delayed Rejection Adaptive Metropolis (DRAM), a Markov Chain Monte Carlo (MCMC) algorithm, to infer the location and intensity of an unknown source. The DRAM results yield a posterior probability distribution for the source's location conditioned on the observed detector count rates. The posterior distribution exhibits regions of high and low probability within the simulated environment identifying potential source locations. In this manner, we can quantify the source location to within at least one of these regions of high probability in the considered cases. Employing these methods, we are able to reduce the space of potential source locations by at least 60%.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Investigating the effects of precise mass measurements of Ru and Pd isotopes on machine learning mass modeling

Atomic masses are a foundational quantity in our understanding of nuclear structure, astrophysics, and fundamental symmetries. The longstanding goal of creating a predictive global model for the binding energy of a nucleus remains a significant challenge, however, and prompts the need for precise measurements of atomic masses to serve as anchor points for model developments. We present precise mass measurements of neutron-rich Ru and Pd isotopes performed at the Californium Rare Isotope Breeder Upgrade facility at Argonne National Laboratory using the Canadian Penning Trap mass spectrometer. The masses of 108 Ru, 110 Ru, and 116 Pd were measured to a relative mass precision $\delta$⁢$m/m$ ≈ 10 -8 via the phase-imaging ion-cyclotron-resonance technique, and represent an improvement of approximately an order of magnitude over previous measurements. Further, these mass data were used in conjunction with the physically interpretable machine learning (PIML) model, which uses a mixture density neural network to model mass excesses via a mixture of Gaussian distributions. The effects of our new mass data on a Bayesian-updating of a PIML model are presented.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Model-Form Epistemic Uncertainty Quantification for Modeling with Differential Equations: Application to Epidemiology

Modeling real-world phenomena to any degree of accuracy is a challenge that the scientific research community has navigated since its foundation. Lack of information and limited computational and observational resources necessitate modeling assumptions which, when invalid, lead to model-form error (MFE). The work reported herein explored a novel method to represent model-form uncertainty (MFU) that combines Bayesian statistics with the emerging field of universal differential equations (UDEs). The fundamental principle behind UDEs is simple: use known equational forms that govern a dynamical system when you have them; then incorporate data-driven approaches – in this case neural networks (NNs) – embedded within the governing equations to learn the interacting terms that were underrepresented. Utilizing epidemiology as our motivating exemplar, this report will highlight the challenges of modeling novel infectious diseases while introducing ways to incorporate NN approximations to MFE. Prior to embarking on a Bayesian calibration, we first explored methods to augment the standard (non-Bayesian) UDE training procedure to account for uncertainty and increase robustness of training. In addition, it is often the case that uncertainty in observations is significant; this may be due to randomness or lack of precision in the measurement process. This uncertainty typically manifests as “noisy” observations which deviate from a true underlying signal. To account for such variability, the NN approximation to MFE is endowed with a probabilistic representation and is updated using available observational data in a Bayesian framework. By representing the MFU explicitly and deploying an embedded, data-driven model, this approach enables an agile, expressive, and interpretable method for representing MFU. In this report we will provide evidence that Bayesian UDEs show promise as a novel framework for any science-based, data-driven MFU representation; while emphasizing that significant advances must be made in the calibration of Bayesian NNs to ensure a robust calibration procedure.

97 MATHEMATICS AND COMPUTING↗

Ensemble models for circuit topology estimation, fault detection and classification in distribution systems

This paper presents a methodology for simultaneous fault detection, classification, and topology estimation for adaptive protection of distribution systems. The methodology estimates the probability of the occurrence of each one of these events by using a hybrid structure that combines three sub-systems, a convolutional neural network for topology estimation, a fault detection based on predictive residual analysis, and a standard support vector machine with probabilistic output for fault classification. The input to all these sub-systems is the local voltage and current measurements. A convolutional neural network uses these local measurements in the form of sequential data to extract features and estimate the topology conditions. The fault detector is constructed with a Bayesian stage (a multitask Gaussian process) that computes a predictive distribution (assumed to be Gaussian) of the residuals using the input. Since the distribution is known, these residuals can be transformed into a Standard distribution, whose values are then introduced into a one-class support vector machine. The structure allows using a one-class support vector machine without parameter cross-validation, so the fault detector is fully unsupervised. Finally, a support vector machine uses the input to perform the classification of the fault types. All three sub-systems can work in a parallel setup for both performance and computation efficiency. In conclusion, we test all three sub-systems included in the structure on a modified IEEE123 bus system, and we compare and evaluate the results with standard approaches.

24 POWER TRANSMISSION AND DISTRIBUTION↗