Final Technical Report: Probabilistic Learning, Dimension Reduction, and Spectral Representations for High-Dimensional Uncertainty Quantification
Final Technical Report
Ghanem, Roger [University of Southern California] ↗
SEARCH · Engineering Papers
Results for “Representation learning”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Assembly via disassembly: A case in machine perceptual development
First results in the effort of learning about representations of objects is presented. The questions attempted to be answered are: What is innate and what must be derived from the environment. The problem is casted in the framework of disassembly of an object into two parts.
Bajcsy, Ruzena K.↗
HDBind: encoding of molecular structure with hyperdimensional binary representations
Traditional methods for identifying “hit” molecules from a large collection of potential drug-like candidates rely on biophysical theory to compute approximations to the Gibbs free energy of the binding interaction between the drug and its protein target. These approaches have a significant limitation in that they require exceptional computing capabilities for even relatively small collections of molecules. Increasingly large and complex state-of-the-art deep learning approaches have gained popularity with the promise to improve the productivity of drug design, notorious for its numerous failures. However, as deep learning models increase in their size and complexity, their acceleration at the hardware level becomes more challenging. Hyperdimensional Computing (HDC) has recently gained attention in the computer hardware community due to its algorithmic simplicity relative to deep learning approaches. The HDC learning paradigm, which represents data with high-dimension binary vectors, allows the use of low-precision binary vector arithmetic to create models of the data that can be learned without the need for the gradient-based optimization required in many conventional machine learning and deep learning methods. This algorithmic simplicity allows for acceleration in hardware that has been previously demonstrated in a range of application areas (computer vision, bioinformatics, mass spectrometery, remote sensing, edge devices, etc.). To the best of our knowledge, our work is the first to consider HDC for the task of fast and efficient screening of modern drug-like compound libraries. We also propose the first HDC graph-based encoding methods for molecular data, demonstrating consistent and substantial improvement over previous work. We compare our approaches to alternative approaches on the well-studied MoleculeNet dataset and the recently proposed LIT-PCBA dataset derived from high quality PubChem assays. We demonstrate our methods on multiple target hardware platforms, including Graphics Processing Units (GPUs) and Field Programmable Gate Arrays (FPGAs), showing at least an order of magnitude improvement in energy efficiency versus even our smallest neural network baseline model with a single hidden layer. Our work thus motivates further investigation into molecular representation learning to develop ultra-efficient pre-screening tools. We make our code publicly available at https://github.com/LLNL/hdbind.
59 BASIC BIOLOGICAL SCIENCES↗
Planning Mars Memory: Learning from the Mer Mission
Knowledge management for space exploration is part of a multi-generational effort at recognizing, preserving and transmitting learning. Each mission should be built on the learning, of both successes and failures, derived from previous missions. Knowledge management begins with learning, and the recognition that this learning has produced knowledge. The Mars Exploration Rover mission provides us with an opportunity to track how learning occurs, how it is recorded, and whether the representations of this learning will be optimally useful for subsequent missions. This paper focuses on the MER science and engineering teams during Rover operations. A NASA team conducted an observational study of the ongoing work and learning of the these teams. Learning occurred in a wide variety of areas: how to run two teams on Mars time for three months; how to use the instruments within the constraints of the martian environment, the deep space network and the mission requirements; how to plan science strategy; how best to use the available software tools. This learning is preserved in many ways. Primarily it resides in peoples memories, to be carried on to the next mission. It is also encoded in stones, in programming sequences, in published reports, and in lessons learned activities, Studying learning and knowledge development as it happens allows us to suggest proactive ways of capturing and using it across multiple missions and generations.
Linde, Charlotte↗
Scientific Discovery with Physics-Informed System Identification (Abbreviated Report)
My fellowship research focused on making physics-based simulations faster and more useful through machine learning. Many problems in science and engineering are governed by partial differential equations, but high-fidelity simulations are often too expensive to run repeatedly. I worked on improving Latent Space Dynamics Identification (LaSDI), a reduced-order modeling framework that compresses large simulation data sets into a smaller representation and then learns how that representation evolves over time. The motivation was to develop reduced models that remain accurate for more challenging systems, especially when predictions must remain reliable over long time intervals or when the underlying dynamics are more complicated than standard methods can easily handle. I also contributed to related work on Quandary, a high-performance software effort for simulation and control of open quantum systems, before focusing primarily on Latent Space Dynamics Identification methods. The main outcomes of the fellowship were two new algorithms (both of which were published), Rollout-LaSDI and Higher-Order LaSDI, together with supporting work on multi-stage Latent Space Dynamics Identification. Rollout-LaSDI improved long-term prediction by training the model to stay accurate over extended time horizons, and Higher-Order LaSDI broadened the method so it could model systems with higher-order time dynamics. My contributions to multistage Latent Space Dynamics Identification also helped show that its later training stages could be simplified without losing effectiveness, and that this behavior held across different model architectures and training strategies. Taken together, these advances improved the accuracy, flexibility, and practical value of reduced-order modeling tools for computational science.
97 MATHEMATICS AND COMPUTING↗
Blueprints for Training Information Bottlenecks for Collider Analyses
Dimensionality reduction is a crucial aspect of data analysis in high energy physics, even if accompanied by information loss. Several methods, including histogram- and kernel-based analyses, are only computationally feasible for low-dimensional data. Furthermore, simulation models used in HEP can often only be validated for low-dimensional data. We provide several blueprints for using machine learning to create low-dimensional data representations (continuous event variables and discrete classification labels) for use in signal discovery and parameter estimation tasks. We also describe how to design the learned representation to facilitate a) searches with unknown model parameters and b) validation of simulation models in data control regions.
43 PARTICLE ACCELERATORS↗
Decode the Workload: Training Deep Learning Models for Efficient Compute Cluster Representation
Monitoring the status of a high throughput computing cluster running computationally intensive production jobs is a crucial yet challenging system administration task due to the complexity of such systems. To this end, we train autoencoders using the Linux kernel CPU metrics of the cluster. Additionally, we explore assisting these models with graph neural networks to share information across threads within a compute node. The models are compared in terms of their ability to: 1) Produce a compressed latent representation that captures the salient features of the input, 2) Detect anomalous activity, and 3) Make distinction between different kinds of jobs run at Jefferson Lab. The goal is to have a robust encoder whose compressed embeddings are used for several downstream tasks. We extend this study further by deploying these models in a human-in-the-loop production-based setting for the anomaly detection task and discuss the associated implementation aspects such as continual learning and the criterion to generate alarms. This study represents a first step in the endeavor towards building self-supervised large-scale foundation models for computing centers.
Mohammed, Ahmed↗
Efficient sampling of free energy landscapes with functions in Sobolev spaces
Molecular simulations of biological and physical phenomena generally involve sampling complicated, rough energy landscapes characterized by multiple local minima. In this work, we introduce a new family of methods for advanced sampling that draw inspiration from functional representations used in machine learning and approximation theory. As shown here, such representations are particularly well suited for learning free energies using artificial neural networks. As a system evolves through phase space, the proposed methods gradually build a model for the free energy as a function of one or more collective variables, from both the frequency of visits to distinct states and generalized force estimates corresponding to such states. Implementation of the methods is relatively simple and, more importantly, for the representative examples considered in this work, they provide computational efficiency gains of up to several orders of magnitude over other widely used simulation techniques.
Approximation theory↗
Streamlining latent spaces in machine learning using moment pooling
Many machine learning applications involve learning a latent representation of data, which is often high-dimensional and difficult to directly interpret. In this work, we propose “moment pooling,” a natural extension of deep sets networks which drastically decreases the latent space dimensionality of these networks while maintaining or even improving performance. Moment pooling generalizes the summation in deep sets to arbitrary multivariate moments, which enables the model to achieve a much higher effective latent dimensionality for a fixed learned latent space dimension. We demonstrate moment pooling on the collider physics task of quark/gluon jet classification by extending energy flow networks (EFNs) to moment EFNs. We find that moment EFNs with latent dimensions as small as 1 perform similarly to ordinary EFNs with higher latent dimension. This small latent dimension allows for the internal representation to be directly visualized and interpreted, which in turn enables the learned internal jet representation to be extracted in closed form. Published by the American Physical Society 2024
Gambhir, Rikab (ORCID:0000000251080448)↗
Bayesian learning with Gaussian processes for low-dimensional representations of time-dependent nonlinear systems
This work presents a data-driven method for learning low-dimensional time-dependent physics-based surrogate models whose predictions are endowed with uncertainty estimates. We use the operator inference approach to model reduction that poses the problem of learning low-dimensional model terms as a regression of state space data and corresponding time derivatives by minimizing the residual of reduced system equations. Standard operator inference models perform well with accurate training data that are dense in time, but producing stable and accurate models when the state data are noisy and/or sparse in time remains a challenge. Another challenge is the lack of uncertainty estimation for the predictions from the operator inference models. Our approach addresses these challenges by incorporating Gaussian process surrogates into the operator inference framework to (1) probabilistically describe uncertainties in the state predictions and (2) procure analytical time derivative estimates with quantified uncertainties. The formulation leads to a generalized least-squares regression and, ultimately, reduced-order models that are described probabilistically with a closed-form expression for the posterior distribution of the operators. The resulting probabilistic surrogate model propagates uncertainties from the observed state data to reduced-order predictions. Furthermore, we demonstrate the method is effective for constructing low-dimensional models of two nonlinear partial differential equations representing a compressible flow and a nonlinear diffusion–reaction process, as well as for estimating the parameters of a low-dimensional system of nonlinear ordinary differential equations representing compartmental models in epidemiology.
Data-driven model reduction↗
Analytical learning and term-rewriting systems
Analytical learning is a set of machine learning techniques for revising the representation of a theory based on a small set of examples of that theory. When the representation of the theory is correct and complete but perhaps inefficient, an important objective of such analysis is to improve the computational efficiency of the representation. Several algorithms with this purpose have been suggested, most of which are closely tied to a first order logical language and are variants of goal regression, such as the familiar explanation based generalization (EBG) procedure. But because predicate calculus is a poor representation for some domains, these learning algorithms are extended to apply to other computational models. It is shown that the goal regression technique applies to a large family of programming languages, all based on a kind of term rewriting system. Included in this family are three language families of importance to artificial intelligence: logic programming, such as Prolog; lambda calculus, such as LISP; and combinatorial based languages, such as FP. A new analytical learning algorithm, AL-2, is exhibited that learns from success but is otherwise quite different from EBG. These results suggest that term rewriting systems are a good framework for analytical learning research in general, and that further research should be directed toward developing new techniques.
Laird, Philip↗
Learning to Identify Airports from Photographs Versus Navigational Maps Presented at Various Orientations
We previously demonstrated: 1) initial learning of navigational maps produced discriminative reaction times (RTs) that were most rapid when the maps were seen in the same orientation as that in which they were initially learned, 2) RTs were significantly reduced with repeated representations, and 3) information learned from the map was not sufficient for all observers to recognize aerial photographs of the same airports. The present study examined the relative effectiveness of initial training on photographs versus maps.
Cohen, Malcolm M.↗
Imaging Modalities Relevant to Intracranial Pressure Assessment in Astronauts
Learning Objectives of this slide presentation are: 1: To review the morphological changes in orbit structures caused by elevated Intracranial Pressure (ICP), and their imaging representation. 2: To learn about the similarities and differences between MRI and sonographic imaging of the eye and orbit. 3: To learn about the role of MRI and sonography in the noninvasive assessment of intracranial pressure in aerospace medicine, and the added benefits from their combined interpretation.
Sargsyan, Ashot E.↗
Multi-head physics-informed neural networks for learning functional priors and uncertainty quantification
In numerous applications, the integration of prior knowledge and historical information is essential, particularly for tasks requiring the solution of ordinary or partial differential equations (ODEs/PDEs) in data-sparse or noisy environments. For instance, achieving accurate solutions to time-dependent PDEs with limited initial condition measurements necessitates an effective strategy for embedding prior knowledge. Hard-parameter sharing architectures in neural networks (NNs) have demonstrated success in both traditional and scientific machine learning domains, facilitating the learning of informative representations. Here, in this study, we introduce a novel, yet efficient, method to enhance physics-informed neural networks (PINNs) by incorporating a multi-head structure that enables the learning of functional priors from both empirical data and governing physical laws. This prior information can then be used to address data sparsity and high-level noise in solving ODE/PDE problems with uncertainty quantification (UQ). The approach, termed Multi-Head PINN (MH-PINN), consists of a shared body NN and multiple head NNs, each corresponding to an individual PINN instance. Our framework for functional prior learning is carried out in two stages: (1) training the MH-PINNs to develop a shared body NN alongside multiple head NNs, and (2) employing these trained head NNs to estimate a prior distribution through a normalizing flow-based density estimator. The learned functional prior can then be applied as a regularization mechanism in deterministic contexts or as an informative prior within a Bayesian inference framework, aiding in the resolution of subsequent ODE/PDE tasks. We evaluate the efficacy of MH-PINNs across five benchmark problems, including a high-dimensional parametric PDE, all characterized by data sparsity or substantial noise levels. Our findings reveal that MH-PINNs deliver accurate solutions and robust UQ, demonstrating adaptability across a range of complex and challenging scenarios.
Bayesian inference↗
Self-supervised and multi-fidelity learning for extended predictive soil spectroscopy
Infrared spectroscopy is a cost-effective, non-destructive, and environmentally benign technology that is increasingly recognized as an important solution for meeting the global demand for soil data. While both near-infrared (NIR) and mid-infrared (MIR) diffuse reflectance spectroscopy enable rapid estimation of soil properties, they present a significant trade-off: NIR offers superior scalability and lower operational costs, whereas MIR provides higher analytical fidelity by capturing fundamental molecular vibrations. In this study, we propose a self-supervised, multi-fidelity learning framework designed to bridge this gap. Our approach leverages large-scale MIR spectral libraries to learn a compact, transferable latent representation, into which NIR spectra are subsequently aligned for downstream prediction. The workflow consists of pretraining a latent model on a large MIR library, adapting the representation using a smaller paired NIR–MIR dataset, and evaluating generalization on an independent external test set. Across a range of chemical and physical soil properties, we found that MIR-derived embeddings improved prediction accuracy relative to baseline models that used raw MIR inputs. Predictions derived from the spectrum conversion (NIR to MIR) task did not match the performance of the original MIR spectra but were similar or superior to predictive performance of NIR-only models, suggesting the unified spectral latent space can effectively leverage the larger and more diverse MIR dataset for prediction of soil properties not well represented in current NIR libraries.
54 ENVIRONMENTAL SCIENCES↗
Learning fuzzy logic control system
The performance of the Learning Fuzzy Logic Control System (LFLCS), developed in this thesis, has been evaluated. The Learning Fuzzy Logic Controller (LFLC) learns to control the motor by learning the set of teaching values that are generated by a classical PI controller. It is assumed that the classical PI controller is tuned to minimize the error of a position control system of the D.C. motor. The Learning Fuzzy Logic Controller developed in this thesis is a multi-input single-output network. Training of the Learning Fuzzy Logic Controller is implemented off-line. Upon completion of the training process (using Supervised Learning, and Unsupervised Learning), the LFLC replaces the classical PI controller. In this thesis, a closed loop position control system of a D.C. motor using the LFLC is implemented. The primary focus is on the learning capabilities of the Learning Fuzzy Logic Controller. The learning includes symbolic representation of the Input Linguistic Nodes set and Output Linguistic Notes set. In addition, we investigate the knowledge-based representation for the network. As part of the design process, we implement a digital computer simulation of the LFLCS. The computer simulation program is written in 'C' computer language, and it is implemented in DOS platform. The LFLCS, designed in this thesis, has been developed on a IBM compatible 486-DX2 66 computer. First, the performance of the Learning Fuzzy Logic Controller is evaluated by comparing the angular shaft position of the D.C. motor controlled by a conventional PI controller and that controlled by the LFLC. Second, the symbolic representation of the LFLC and the knowledge-based representation for the network are investigated by observing the parameters of the Fuzzy Logic membership functions and the links at each layer of the LFLC. While there are some limitations of application with this approach, the result of the simulation shows that the LFLC is able to control the angular shaft position of the D.C. motor. Furthermore, the LFLC has better performance in rise time, settling time and steady state error than to the conventional PI controller. This abstract accurately represents the content of the candidate's thesis. I recommend its publication.
Lung, Leung Kam↗
Assuring Intelligent Systems: Contingency Management for UAS
Unmanned aircraft systems (UAS) collaborate with humans to operate in diverse, safety-critical applications. However, assurance technologies need to be integrated into the design process in order to guarantee safe behavior, thereby enabling UAS operations in the National Airspace System (NAS). In this paper, formal methods are integrated with learning-enabled systems representations. The generation and representation of knowledge are captured via monadic second-order logic rules in the cognitive architecture Soar. These rules are translated into timed automata, and a proof of correctness for the translation is provided so that safety and liveness properties can be checked in the formal verification environment Uppaal. This approach is agnostic to the learning mechanism used to generate the learned rules (e.g., chunking, etc.). An example of a fault-tolerant, learning-enabled UAS deciding which of four contingency procedures to execute under a lost link scenario while overflying an urban area is used to illustrate the approach.
Intelligent Systems↗
When ancient numerical demons meet physics-informed machine learning: adjoint-based gradients for implicit differentiable modeling
Recent advances in differentiable modeling, a genre of physics-informed machine learning that trains neural networks (NNs) together with process-based equations, have shown promise in enhancing hydrological models' accuracy, interpretability, and knowledge-discovery potential. Current differentiable models are efficient for NN-based parameter regionalization, but the simple explicit numerical schemes paired with sequential calculations (operator splitting) can incur numerical errors whose impacts on models' representation power and learned parameters are not clear. Implicit schemes, however, cannot rely on automatic differentiation to calculate gradients due to potential issues of gradient vanishing and memory demand. Here we propose a “discretize-then-optimize” adjoint method to enable differentiable implicit numerical schemes for the first time for large-scale hydrological modeling. The adjoint model demonstrates comprehensively improved performance, with Kling–Gupta efficiency coefficients, peak-flow and low-flow metrics, and evapotranspiration that moderately surpass the already-competitive explicit model. Therefore, the previous sequential-calculation approach had a detrimental impact on the model's ability to represent hydrological dynamics. Furthermore, with a structural update that describes capillary rise, the adjoint model can better describe baseflow in arid regions and also produce low flows that outperform even pure machine learning methods such as long short-term memory networks. The adjoint model rectified some parameter distortions but did not alter spatial parameter distributions, demonstrating the robustness of regionalized parameterization. Despite higher computational expenses and modest improvements, the adjoint model's success removes the barrier for complex implicit schemes to enrich differentiable modeling in hydrology.
58 GEOSCIENCES↗