Stein variational inference and sparsification for uncertainty quantification of physical neural networks
Explore the source record for details and available documents.
Jones, Reese Edward [Sandia National Laboratories ↗
SEARCH · Engineering Papers
Results for “Variational inference”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Decentralized Black-Box Variational Inference for Bayesian Learning on Sensor Networks
Explore the source record for details and available documents.
Engineering - Electronic and electrical engineerin↗
Probabilistic Nanomagnetic Memories for Uncertain and Robust Machine Learning
This project evaluated the use of emerging spintronic memory devices for robust and efficient variational inference schemes. Variational inference (VI) schemes, which constrain the distribution for each weight to be a Gaussian distribution with a mean and standard deviation, are a tractable method for calculating posterior distributions of weights in a Bayesian neural network such that this neural network can also be trained using the powerful backpropagation algorithm. Our project focuses on domain-wall magnetic tunnel junctions (DW-MTJs), a powerful multi-functional spintronic synapse design that can achieve low power switching while also opening the pathway towards repeatable, analog operation using fabricated notches. Our initial efforts to employ DW-MTJs as an all-in-one stochastic synapse with both a mean and standard deviation didn’t end up meeting the quality metrics for hardware-friendly VI. In the future, new device stacks and methods for expressive anisotropy modification may make this idea still possible. However, as a fall back that immediately satisfies our requirements, we invented and detailed how the combination of a DW-MTJ synapse encoding the mean and a probabilistic Bayes-MTJ device, programmed via a ferroelectric or ionically modifiable layer, can robustly and expressively implement VI. This design includes a physics-informed small circuit model, that was scaled up to perform and demonstrate rigorous uncertainty quantification applications, up to and including small convolutional networks on a grayscale image classification task, and larger (Residual) networks implementing multi-channel image classification. Lastly, as these results and ideas all depend upon the idea of an inference application where weights (spintronic memory states) remain non-volatile, the retention of these synapses for the notched case was further interrogated. These investigations revealed and emphasized the importance of both notch geometry and anisotropy modification in order to further enhance the endurance of written spintronic states. In the near future, these results will be mapped to effective predictions for room temperature and elevated operation DW-MTJ memory retention, and experimentally verified when devices become available.
77 NANOSCIENCE AND NANOTECHNOLOGY↗
Inference of variations in the gravity field from satellite-to-satellite range rate
An analytic scheme for inferring variations of the gravity field from satellite-to-satellite range rate (low-low) is developed. As a test, it is applied to a pair of satellites in polar orbit, at altitude 160 km and spacing 100 km, with 72 data points per revolution. An assumed gravity field of tesseral spherical harmonics up to the eighth degree is completely recovered in three iterations over 64 revolutions. It is apparent that data points at regular intervals enable the use of data analysis techniques that avoid massive matrix inversions.
Kaula, W. M.↗
B-PINNs: Bayesian Physics-informal Neural Networks for Forward and Inverse PDE Problems with Noisy Data
We propose a Bayesian physics-informed neural network (B-PINN) to solve both forward and inverse nonlinear problems described by partial differential equations (PDEs) and noisy data. In this Bayesian framework, the Bayesian neural network (BNN) combined with a PINN for PDEs serves as the prior while the Hamiltonian Monte Carlo (HMC) or the variational inference (VI) could serve as an estimator of the posterior. B-PINNs make use of both physical laws and scattered noisy measurements to provide predictions and quantify the aleatoric uncertainty arising from the noisy data in the Bayesian framework. Compared with PINNs, in addition to uncertainty quantification, B-PINNs obtain more accurate predictions in scenarios with large noise due to their capability of avoiding overfitting. We conduct a systematic comparison between the two different approaches for the B-PINNs posterior estimation (i.e., HMC or VI), along with dropout used for quantifying uncertainty in deep neural networks. Our experiments show that HMC is more suitable than VI with mean field Gaussian approximation for the B-PINNs posterior estimation, while dropout employed in PINNs can hardly provide accurate predictions with reasonable uncertainty. Finally, we replace the BNN in the prior with a truncated Karhunen-Loève (KL) expansion combined with HMC or a deep normalizing flow (DNF) model as posterior estimators. The KL is as accurate as BNN and much faster but this framework cannot be easily extended to high-dimensional problems unlike the BNN based framework.
Non linear PDEs, Noisy data, Bysian physics inform↗
Jensen–Shannon divergence based novel loss functions for Bayesian neural networks
Bayesian neural networks (BNNs) are state-of-the-art machine learning methods that can naturally regularize and systematically quantify uncertainties using their stochastic parameters. Kullback–Leibler (KL) divergence-based variational inference used in BNNs suffer from unstable optimization and challenges in approximating light-tailed posteriors due to the unbounded nature of the KL divergence. To resolve these issues, we formulate a novel loss function for BNNs based on a new modification to the generalized Jensen–Shannon (JS) divergence, which is bounded. In addition, we propose a Geometric JS divergence-based loss, which is computationally efficient since it can be evaluated analytically. We found that the JS divergence-based variational inference is intractable, and hence employed a constrained optimization framework to formulate these losses. Our theoretical analysis and empirical experiments on multiple regression and classification data sets suggest that the proposed losses perform better than the KL divergence-based loss, especially when the data sets are noisy or biased. Specifically, there are approximately 5% and 8% improvements in accuracy for a noise-added CIFAR-10 dataset and a regression dataset, respectively. There is about 13% reduction in false negative predictions of a biased histopathology dataset. Additionally, we quantify and compare the uncertainty metrics for the regression and classification tasks.
97 MATHEMATICS AND COMPUTING↗
Quasi-Newton Variational Bayes v.3.0
SAND2024-09052O Quasi-Newton Variational Bayes (QNVB) is a training algorithm that performs high-dimensional variational inference on machine learning models. It provides mechanisms to calibrate and control model uncertainty during training. The PyTorch implementation of projective integral updates for Gaussian mean-fields supports the manuscript, "Projective Integral Updates for High-Dimensional Variational Inference." Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Duersch, Jed↗
Prime VI
SAND2025-03757O Prime VI is a distribution-of-disease outbreak model calibration code based on variational inference. It accompanies a publication for submission to Statistics in Medicine journal, and the code will be maintained for open-source use on Sandia's GitLab. The software provides methods for calibrating an epidemiological model to measured case-count data for a multitude of correlated spatial regions. The code solves a Bayesian inverse problem for model calibration where the posterior over-model parameters are approximated through a custom implementation of variational inference. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.
Safta, Cosmin↗
Learning genetic perturbation effects with variational causal inference
Advances in sequencing technologies have enhanced the understanding of gene regulation in cells. In particular, Perturb-seq has enabled high-resolution profiling of the transcriptomic response to genetic perturbations at the single-cell level. This understanding has implications in functional genomics and potentially for identifying therapeutic targets. Various computational models have been developed to predict perturbational effects. While deep learning models excel at interpolating observed perturbational data, they tend to overfit in the lack of enough data and may not generalize well to unseen perturbations. In contrast, mechanistic models, such as linear causal models based on gene regulatory networks, hold greater potential for extrapolation, as they encapsulate regulatory information that can predict responses to unseen perturbations. However, their application has been limited to small studies due to overly simplistic assumptions, making them less effective in handling noisy, large-scale single-cell data. We propose a hybrid approach that combines a mechanistic causal model with variational deep learning, termed Single Cell Causal Variational Autoencoder (SCCVAE). The mechanistic model employs a learned regulatory network to represent perturbational changes as shift interventions that propagate through the learned network. SCCVAE integrates this mechanistic causal model into a variational autoencoder, generating rich, comprehensive transcriptomic responses. Our results indicate that SCCVAE exhibits superior performance over current state-of-the-art baselines for extrapolating to predict unseen perturbational responses. Additionally, for the observed perturbations, the latent space learned by SCCVAE allows for the identification of functional perturbation modules and simulation of single-gene knockdown experiments of varying penetrance, presenting a robust tool for interpreting and interpolating perturbational responses at the single-cell level.
59 BASIC BIOLOGICAL SCIENCES↗
Extending Parsimonious Bayesian Inference
Parsimonious Bayesian inference is a theoretical framework for efficient data assimilation that seeks to balance increased consistency between predictions and training data against corresponding increases in model complexity. Within this framework, over-training is understood as optimization that encodes excessive information within model parameters while only achieving small improvements between predictions and training data. This project aims to develop practical methods of limiting excess model information during optimization. One key observation is that practical heuristics for parsimonious learning in high-dimensions must balance expressivity, i.e. the ability of the model to capture diverse predictions with only a few non-zero parameters, against discoverability, i.e. the ability to train the model with gradient-based optimization and drive parameters to low information states. As such, we developed logical activation functions that are able to adaptively approximate arbitrary truth tables that define Boolean logic operations within a probabilistic framework. These functions have demonstrated the ability to learn exclusive disjunction (XOR) and conditioned disjunction (if [condition] then [result_if_true] else [result_if_false]) within a single layer of a neural network. To efficiently exploit these activation functions to drive parsimonious learning required several other advances within the domain of variational inference. The most efficient form of complexity suppression is structured sparsification, driving most model parameters to zero while achieving the structural coherence among nonzeros needed for bandwidth reduction. Such models are not only far more efficient at suppressing information-theoretic complexity, they also reduce the other forms of complexity (computations, communication, storage, and the number of dependencies needed to evaluate predictions). Aiming to support enhanced sparsification, this project examined new approaches to high-dimensional variational inference that allow us to calibrate and control parameter uncertainty during optimization. By identifying which parameters can sustain sparsifying perturbations with little impact on prediction quality, we can develop better pruning strategies by framing them as approximate Bayesian inference. These advances also open paths to mitigate concerns with deploying advanced learning methods in resource-constrained environments, such as running models on power-limited or communication-limited devices.
97 MATHEMATICS AND COMPUTING↗
Adapting Reinforcement Learning Treatment Policies Using Limited Data to Personalize Critical Care
We report reinforcement learning (RL) demonstrates promise for developing effective treatment policies in critical care settings. However, existing RL methods often require large and comprehensive patient data sets and do not readily lend themselves to settings in which certain patient subpopulations are severely underrepresented. In this study, we develop a new method, noisy Bayesian policy updates (NBPU), for selecting high-performing reinforcement learning–based treatment policies for underrepresented patient subpopulations using limited observations. Our method uses variational inference to learn a probability distribution over treatment policies based on a reference patient subpopulation for which sufficient data are available. It then exploits limited data from an underrepresented patient subpopulation to update this probability distribution and adapts its recommendations to this subpopulation. We demonstrate our method’s utility on a data set of ICU patients receiving intravenous blood anticoagulant medication. Our results show that NBPU outperforms state-of-the-art methods in terms of both selecting effective treatment policies for patients with nontypical clinical characteristics and predicting the corresponding policies’ performance for these patients.
60 APPLIED LIFE SCIENCES↗
A differentiable simulation package for performing inference of synchrotron-radiation-based diagnostics
The direction of particle accelerator development is ever-increasing beam quality, currents and repetition rates. This poses a challenge to traditional diagnostics that directly intercept the beam due to the mutual destruction of both the beam and the diagnostic. An alternative approach is to infer beam parameters non-invasively from the synchrotron radiation emitted in bending magnets. However, inferring the beam distribution from a measured radiation pattern is a complex and computationally expensive task. To address this challenge we present SYRIPY ( SYnchrotron Radiation In PYthon ), a software package intended as a tool for performing inference of synchrotron-radiation-based diagnostics. SYRIPY has been developed using PyTorch , which makes it both differentiable and able to leverage the high performance of GPUs, two vital characteristics for performing statistical inference. The package consists of three modules: a particle tracker, Lienard–Wiechert solver and Fourier optics propagator, allowing start-to-end simulation of synchrotron radiation detection to be carried out. SYRIPY has been benchmarked against SRW , the prevalent numerical package in the field, showing good agreement and up to a 50× speed improvement. Finally, we have demonstrated how SYRIPY can be used to perform Bayesian inference of beam parameters using stochastic variational inference.
43 PARTICLE ACCELERATORS↗
Simultaneous global and local clustering in multiplex networks with covariate information
Understanding both global and layer-specific group structures is useful for uncovering complex patterns in networks with multiple interaction types. In this work, we introduce a new model, the hierarchical multiplex stochastic blockmodel, which simultaneously detects communities within individual layers of a multiplex network while inferring a global node clustering across the layers. A stochastic blockmodel is assumed in each layer, with probabilities of layer-level group memberships determined by a node’s global group assignment. Our model uses a Bayesian framework, employing a probit stick-breaking process to construct node-specific mixing proportions over a set of shared Griffiths–Engen–McCloseky distributions. These proportions determine layer-level community assignment, allowing for an unknown and varying number of groups across layers, while incorporating nodal covariate information to inform the global clustering. We propose a scalable variational inference procedure with parallelisable updates for application to large networks. Extensive simulation studies demonstrate our model’s ability to accurately recover both global and layer-level clusters in complicated settings, and applications to real data showcase the model’s effectiveness in uncovering interesting latent network structure.
community detection↗
Unsupervised physics-informed disentanglement of multimodal data
Here, we introduce physics-informed multimodal autoencoders (PIMA) - a variational inference framework for discovering shared information in multimodal datasets. Individual modalities are embedded into a shared latent space and fused through a product-of-experts formulation, enabling a Gaussian mixture prior to identify shared features. Sampling from clusters allows cross-modal generative modeling, with a mixture-of-experts decoder that imposes inductive biases from prior scientific knowledge and thereby imparts structured disentanglement of the latent space. This approach enables cross-modal inference and the discovery of features in high-dimensional heterogeneous datasets. Consequently, this approach provides a means to discover fingerprints in multimodal scientific datasets and to avoid traditional bottlenecks related to high-fidelity measurement and characterization of scientific datasets.
97 MATHEMATICS AND COMPUTING↗
Multi-Modal Bayesian Neural Network Surrogates with Conjugate Last-Layer Estimation
As data collection and simulation capabilities advance, multi-modal learning, the task of learning from multiple modalities and sources of data, is becoming an increasingly important area of research. Surrogate models that learn from data of multiple auxiliary modalities to support the modeling of a highly expensive quantity of interest have the potential to aid outer loop applications such as optimization, inverse problems, or sensitivity analyses when multi-modal data are available. We develop two multi-modal Bayesian neural network surrogate models and leverage conditionally conjugate distributions in the last layer to estimate model parameters using stochastic variational inference (SVI). We provide a method to perform this conjugate SVI estimation in the presence of partially missing observations. Here, we demonstrate improved prediction accuracy and uncertainty quantification compared to unimodal surrogate models for both scalar and time series data.
97 MATHEMATICS AND COMPUTING↗
Spike-and-Slab Shrinkage Priors for Structurally Sparse Bayesian Neural Networks
Network complexity and computational efficiency have become increasingly significant aspects of deep learning. Sparse deep learning addresses these challenges by recovering a sparse representation of the underlying target function by reducing heavily overparameterized deep neural networks. Specifically, deep neural architectures compressed via structured sparsity (e.g., node sparsity) provide low-latency inference, higher data throughput, and reduced energy consumption. In this article, we explore two well-established shrinkage techniques, Lasso and Horseshoe, for model compression in Bayesian neural networks (BNNs). To this end, we propose structurally sparse BNNs, which systematically prune excessive nodes with the following: 1) spike-and-slab group Lasso (SS-GL) and 2) SS group Horseshoe (SS-GHS) priors, and develop computationally tractable variational inference, including continuous relaxation of Bernoulli variables. We establish the contraction rates of the variational posterior of our proposed models as a function of the network topology, layerwise node cardinalities, and bounds on the network weights. Furthermore, we empirically demonstrate the competitive performance of our models compared with the baseline models in prediction accuracy, model compression, and inference latency.
97 MATHEMATICS AND COMPUTING↗
Geomagnetic storm fields near a synchronous satellite.
An apparent early recovery of the main phase of geomagnetic storms at the distance of the synchronous satellite is examined in terms of changing electric current distributions in the magnetosphere during magnetic storms. It is suggested that a rapid recession of the edge of the plasma sheet (after the advance toward the earth during an early epoch of the main phase) is partly responsible for the early recovery. Relevant plasma sheet variations during geomagnetic storms are found to be in agreement with the inferred variations.
Kawasaki, K.↗
Comparison of atomic oxygen measurements by incoherent scatter and satellite-borne mass spectrometer techniques
Atomic oxygen densities determined by the incoherent scatter technique are compared to densities deduced from satellite-borne mass spectrometer measurements and are found to agree within experimental error. The diurnal variations inferred from the incoherent scatter measurements do show, however, some departure from diurnal variations found by modeling the mass spectrometer results. Some implications of these departures are briefly discussed.
Hedin, A. E.↗