Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Bayesian learning”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 541 records · Page 30
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Evaluating the Potential and Challenges of an Uncertainty Quantification Method for Long Short–Term Memory Models for Soil Moisture Predictions

Recently, recurrent deep networks have shown promise to harness newly available satellite–sensed data for long–term soil moisture projections. However, to be useful in forecasting, deep networks must also provide uncertainty estimates. Here we evaluated Monte Carlo dropout with an input–dependent data noise term (MCD+N), an efficient uncertainty estimation framework originally developed in computer vision, for hydrologic time series predictions. MCD+N simultaneously estimates a heteroscedastic input–dependent data noise term (a trained error model attributable to observational noise) and a network weight uncertainty term (attributable to insufficiently constrained model parameters). Although MCD+N has appealing features, many heuristic approximations were employed during its derivation, and rigorous evaluations and evidence of its asserted capability to detect dissimilarity were lacking. To address this, we provided an in–depth evaluation of the scheme's potential and limitations. We showed that for reproducing soil moisture dynamics recorded by the Soil Moisture Active Passive (SMAP) mission, MCD+N indeed gave a good estimate of predictive error, provided that we tuned a hyperparameter and used a representative training data set. The input–dependent term responded strongly to observational noise, while the model term clearly acted as a detector for physiographic dissimilarity from the training data, behaving as intended. However, when the training and test data were characteristically different, the input–dependent term could be misled, undermining its reliability. Additionally, due to the data–driven nature of the model, data noise also influences network weight uncertainty, and therefore the two uncertainty terms are correlated. Altogether, this approach has promise, but care is needed to interpret the results.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Optimizing training trajectories in variational autoencoders via latent Bayesian optimization approach *

Unsupervised and semi-supervised ML methods such as variational autoencoders (VAE) have become widely adopted across multiple areas of physics, chemistry, and materials sciences due to their capability in disentangling representations and ability to find latent manifolds for classification and/or regression of complex experimental data. Like other ML problems, VAEs require hyperparameter tuning, e.g. balancing the Kullback–Leibler and reconstruction terms. However, the training process and resulting manifold topology and connectivity depend not only on hyperparameters, but also their evolution during training. Because of the inefficiency of exhaustive search in a high-dimensional hyperparameter space for the expensive-to-train models, here we have explored a latent Bayesian optimization (zBO) approach for the hyperparameter trajectory optimization for the unsupervised and semi-supervised ML and demonstrated for joint-VAE with rotational invariances. We have demonstrated an application of this method for finding joint discrete and continuous rotationally invariant representations for modified national institute of standards and technology database (MNIST) and experimental data of a plasmonic nanoparticles material system. The performance of the proposed approach has been discussed extensively, where it allows for any high dimensional hyperparameter trajectory optimization of other ML models.

42 ENGINEERING↗
DOE OSTI2015

Forensic analysis of the microbiome of phones and shoes

Background: Microbial interaction between human-associated objects and the environments we inhabit may have forensic implications, and the extent to which microbes are shared between individuals inhabiting the same space may be relevant to human health and disease transmission. In this study, two participants sampled the front and back of their cell phones, four different locations on the soles of their shoes, and the floor beneath them every waking hour over a 2-day period. A further 89 participants took individual samples of their shoes and phones at three different scientific conferences. Results: Samples taken from different surface types maintained significantly different microbial community structures. The impact of the floor microbial community on that of the shoe environments was strong and immediate, as evidenced by Procrustes analysis of shoe replicates and significant correlation between shoe and floor samples taken at the same time point. Supervised learning was highly effective at determining which participant had taken a given shoe or phone sample, and a Bayesian method was able to determine which participant had taken each shoe sample based entirely on its similarity to the floor samples. Both shoe and phone samples taken by conference participants clustered into distinct groups based on location, though much more so when an unweighted distance metric was used, suggesting sharing of low-abundance microbial taxa between individuals inhabiting the same space. In conclusion, correlations between microbial community sources and sinks allow for inference of the interactions between humans and their environment.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2020

A Survey of Constrained Gaussian Process: Approaches and Implementation Challenges

Gaussian process regression is a popular Bayesian framework for surrogate modeling of expensive data sources. As part of a larger effort in scientific machine learning, many recent works have incorporated physical constraints or other a priori information within Gaussian process regression to supplement limited data and regularize the behavior of the model. We provide an overview and survey of several classes of Gaussian process constraints, including positivity or bound constraints, monotonicity and convexity constraints, differential equation constraints provided by linear PDEs, and boundary condition constraints. We compare the strategies behind each approach as well as the differences in implementation, concluding with a discussion of the computational challenges introduced by constraints.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Constrained Bayesian Optimization of Criticality Experiments [Slides]

The design of criticality experiments is typically an iterative process that employs a Monte Carlo transport code. The goal is to find a design that optimizes some variable, like the sensitivity of a response to a cross section, while simultaneously ensuring criticality. The high fidelity of the Monte Carlo code is a great asset, but it makes exploring the design space computationally expensive. Herein, we present how a constrained Bayesian optimization algorithm can be used to efficiently design a criticality experiment. It uses Gaussian processes as a surrogate model to probe the design space and to reduce the number of code executions that are needed to find the optimum. We demonstrate constrained Bayesian optimization with a Pu-239/polyethylene solution system and a TEX experiment that is designed for criticality safety validation of a nuclear waste model at the Hanford Site. For both systems, a global optimum was found within 75 Monte Carlo simulations.

42 ENGINEERING↗
DOE OSTI2020

Constrained Bayesian Optimization of Criticality Experiments at LLNL [Slides]

The design of criticality experiments is typically an iterative process that employs a Monte Carlo transport code. The goal is to find a design that optimizes some variable, like the sensitivity of a response to a cross section, while simultaneously ensuring criticality. The high fidelity of the Monte Carlo code is a great asset, but it makes exploring the design space computationally expensive. Herein, we present how a constrained Bayesian optimization algorithm can be used to efficiently design a criticality experiment. It uses Gaussian processes as a surrogate model to probe the design space and to reduce the number of code executions that are needed to find the optimum. We demonstrate constrained Bayesian optimization with a Pu-239/polyethylene solution system and a TEX experiment that is designed for critical ity safety validation of a nuclear waste model at the Hanford Site. For both systems, a global optimum was found within 75 Monte Carlo simulations.

42 ENGINEERING↗
DOE OSTI2024

Human limits in machine learning: prediction of potato yield and disease using soil microbiome data

Abstract Background The preservation of soil health is a critical challenge in the 21st century due to its significant impact on agriculture, human health, and biodiversity. We provide one of the first comprehensive investigations into the predictive potential of machine learning models for understanding the connections between soil and biological phenotypes. We investigate an integrative framework performing accurate machine learning-based prediction of plant performance from biological, chemical, and physical properties of the soil via two models: random forest and Bayesian neural network. Results Prediction improves when we add environmental features, such as soil properties and microbial density, along with microbiome data. Different preprocessing strategies show that human decisions significantly impact predictive performance. We show that the naive total sum scaling normalization that is commonly used in microbiome research is one of the optimal strategies to maximize predictive power. Also, we find that accurately defined labels are more important than normalization, taxonomic level, or model characteristics. ML performance is limited when humans can’t classify samples accurately. Lastly, we provide domain scientists via a full model selection decision tree to identify the human choices that optimize model prediction power. Conclusions Our study highlights the importance of incorporating diverse environmental features and careful data preprocessing in enhancing the predictive power of machine learning models for soil and biological phenotype connections. This approach can significantly contribute to advancing agricultural practices and soil health management.

Aghdam, Rosa↗
DOE OSTI2025

A neural-network-enhanced parameter-varying framework for multi-objective model predictive control applied to buildings

Management of the electrical grid is becoming more complex due to the increased penetration of alternative energy generation technologies and a broadening diversity of electric loads. This complexity creates challenges in balancing demand and generation that can increase the potential for grid instabilities. One effective way to address this issue is to leverage previously unexploited demand flexibility through advanced control strategies. In this work, we propose an advanced control method, called adaptive neural parameter-varying model predictive control (ANPV-MPC), to control the temperature and energy consumption of a building via its Heating, Ventilation, and Air Conditioning system. ANPV-MPC combines key ideas in parameter-varying control, adaptive control, and online learning strategies to bridge the gap between computationally efficient linear model predictive control and more accurate nonlinear model predictive control. The novelty in ANPV-MPC is the use of a physics-inspired Bayesian neural network to estimate the coefficients of the parameter-varying linear control model. The Bayesian neural network additionally provides uncertainty estimates, triggering online training to capture evolving building system conditions. We show that ANPV-MPC can approximate the building system dynamics with a 28.39% higher accuracy than traditional linear model predictive control, resulting in 36.23% better control performance without increasing complexity of the optimal control problem. ANPV-MPC also adapts in real time to previously unseen conditions using online learning, further improving its performance.

24 POWER TRANSMISSION AND DISTRIBUTION↗
NASA NTRS2004

Putting Priors in Mixture Density Mercer Kernels

This paper presents a new methodology for automatic knowledge driven data mining based on the theory of Mercer Kernels, which are highly nonlinear symmetric positive definite mappings from the original image space to a very high, possibly infinite dimensional feature space. We describe a new method called Mixture Density Mercer Kernels to learn kernel function directly from data, rather than using predefined kernels. These data adaptive kernels can en- code prior knowledge in the kernel using a Bayesian formulation, thus allowing for physical information to be encoded in the model. We compare the results with existing algorithms on data from the Sloan Digital Sky Survey (SDSS). The code for these experiments has been generated with the AUTOBAYES tool, which automatically generates efficient and documented C/C++ code from abstract statistical model specifications. The core of the system is a schema library which contains template for learning and knowledge discovery algorithms like different versions of EM, or numeric optimization methods like conjugate gradient methods. The template instantiation is supported by symbolic- algebraic computations, which allows AUTOBAYES to find closed-form solutions and, where possible, to integrate them into the code. The results show that the Mixture Density Mercer-Kernel described here outperforms tree-based classification in distinguishing high-redshift galaxies from low- redshift galaxies by approximately 16% on test data, bagged trees by approximately 7%, and bagged trees built on a much larger sample of data by approximately 2%.

Srivastava, Ashok N.↗
DOE OSTI2021

A Multi-Objective Bayesian Optimization Approach Using the Weighted Tchebycheff Method

Abstract Bayesian optimization (BO) is a low-cost global optimization tool for expensive black-box objective functions, where we learn from prior evaluated designs, update a posterior surrogate Gaussian process model, and select new designs for future evaluation using an acquisition function. This research focuses upon developing a BO model with multiple black-box objective functions. In the standard multi-objective (MO) optimization problem, the weighted Tchebycheff method is efficiently used to find both convex and non-convex Pareto frontiers. This approach requires knowledge of utopia values before we start optimization. However, in the BO framework, since the functions are expensive to evaluate, it is very expensive to obtain the utopia values as a prior knowledge. Therefore, in this paper, we develop a MO-BO framework where we calibrate with multiple linear regression (MLR) models to estimate the utopia value for each objective as a function of design input variables; the models are updated iteratively with sampled training data from the proposed MO-BO. These iteratively estimated mean utopia values are used to formulate the weighted Tchebycheff MO acquisition function. The proposed approach is implemented in two numerical test examples and one engineering design problem of optimizing thin tube geometries under constant loading of temperature and pressure, with minimizing the risk of creep-fatigue failure and design cost, along with risk-based and manufacturing constraints. Finally, the model accuracy with frequentist, Bayesian and without MLR-based calibration are compared to true Pareto solutions.

Engineering↗
DOE OSTI2021

MADLens, a python package for fast and differentiable non-Gaussian lensing simulations

Here, we present MADLens a python package for producing non-Gaussian lensing convergence maps at arbitrary source redshifts with unprecedented precision. MADLens is designed to achieve high accuracy while keeping computational costs as low as possible. A MADLens simulation with only particles produces convergence maps whose power agrees with theoretical lensing power spectra up to within the accuracy limits of HaloFit. This is made possible by a combination of a highly parallelizable particle-mesh algorithm, a sub-evolution scheme in the lensing projection, and a machine-learning inspired sharpening step. Further, MADLens is fully differentiable with respect to the initial conditions of the underlying particle-mesh simulations and a number of cosmological parameters. These properties allow MADLens to be used as a forward model in Bayesian inference algorithms that require optimization or derivative-aided sampling. Another use case for MADLens is the production of large, high resolution simulation sets as they are required for training novel deep-learning-based lensing analysis tools. We make the MADLens package publicly available under a Creative Commons License

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2021

A novel machine learning based identification of potential adopter of rooftop solar photovoltaics

With the proliferation of rooftop solar photovoltaic installations, there is a need to proactively predict consumer potential for solar photovoltaic adoption, for improved electric utility planning and operation. Traditional analytical modeling approaches are limited to a few survey features and a larger part of the survey would remain untouched by the decision model. This article presents a novel, data-driven modeling approach that strategically prunes a large set of consumer profile features using a machine learning framework to train a model for predicting potential solar adoption. The approach utilizes the Gradient Boosting Decision Tree model through a Light Gradient Boosting framework that improves significantly over the poor prediction accuracy of the existing approaches. Model training using focal-loss based supervision is used to overcome the difficulty in identifying the potential adopters that is inherent in conventional data-driven models. In addition, to overcome possible data sparsity in a limited survey sample, a Generative Adversarial Network is presented to create synthetic user samples and its effectiveness on model performance is assessed. A Bayesian optimization approach is used to systematically arrive at the hyperparameters of the proposed model. Validation of the presented approach on a survey data collected by the National Rural Electric Cooperative Association in Virginia in 2018 demonstrates the excellent predictive capability of the machine learning based approach to modeling solar adoption reliably.

14 SOLAR ENERGY↗
DOE OSTI2025

Autonomous organic synthesis for redox flow batteries via flexible batch Bayesian optimization

Traditional trial-and-error methods for materials discovery are inefficient to meet the urgent demands posed by the rapid progression of climate change. This urgency has driven the increasing interest in integrating robotics and machine learning into materials research to accelerate experimental learning. However, idealized decision-making frameworks to achieve maximum sampling efficiency are not always compatible with high-throughput experimental workflows inside a laboratory. For multi-step chemical processes, differences in hardware capacities can complicate the digital framework by introducing constraints on the maximum number of samples in each step of the experiment, hence causing varying batch sizes in variable selection within the same batch. Therefore, designing flexible sampling algorithms is necessary to accommodate the multi-step synthesis with practical constraints unique to each high-throughput workflow. In this work, we designed and employed three strategies on a high-throughput robotic platform to optimize the sulfonation reaction of redox-active molecules used in flow batteries. Our strategies adapt to the multi-step experimental workflow, where their formulation and heating steps are separate, causing varying batch size requirements. By strategically sampling using clustering and mixed-variable batch Bayesian optimization, we were able to iteratively identify optimal conditions that maximize the yields. Our work presents a flexible approach that allows tailoring the machine learning decision-making to suit the practical constraints in individual high-throughput experimental platforms, followed by performing resource-efficient yield optimization using available open-source Python libraries.

Tamura, Clara [Univ. of Washington, Seattle, WA (U↗
DOE OSTI2025

Comparison of tungsten versus molybdenum for double shell capsules using machine learning design optimization

Double shell targets are an alternative ignition platform for inertial confinement fusion. One design consideration for double shell targets is the choice of inner shell material to help trap radiation emitted by the hot fuel to aid ignition. Materials such as molybdenum and tungsten are of interest for the inner shell layer of the targets. While molybdenum has a lower density that could inhibit instability growth and allow for radiography and code benchmarking, tungsten has a higher density that could provide more compression and confinement. These tradeoffs have been explored using optimized designs for each material. Our previous work [Vazirani et al., “Coupling 1D xRAGE simulations with machine learning for graded inner shell design optimization in double shell capsules,” Phys. Plasmas 28, 122709 (2021); Vazirani et al., “Coupling multi-fidelity xRAGE with machine learning for graded inner shell design optimization in double shell capsules,” Phys. Plasmas 30, 062704 (2023); and Vazirani et al., “Bayesian batch optimization for molybdenum versus tungsten inertial confinement fusion double shell target design,” Stat. Anal. Data Min. 17, e11698 (2024)] resulted in a multi-fidelity Bayesian optimization framework to find yield-optimized double shell target geometries. By leveraging simulations of varying fidelities (one-dimensional and two-dimensional) to inform one another, the multi-fidelity optimization was able to optimize a design in the highest fidelity with significantly fewer simulations than would be used in a systematic parameter scan. In this work, we apply the multi-fidelity Bayesian optimization to explore the optimized designs of double shell targets with molybdenum and tungsten inner shells as well as the physics producing the high performing implosions. A physics exploration of all the simulations used in this study shows trends in designs that contribute to high yields, ion temperatures, and fuel areal densities. Comparison of molybdenum and tungsten simulations shows that they can produce similar implosion conditions with different geometries, which would be important to study in experiments. Graded density layers produce varying performances with the two materials but continue to be of interest for future studies along with studies of doped inner shell materials and applied surface roughness.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2023

Adaptive Discovery and Mixed-Variable Optimization of Next Generation Synthesizable Microelectronic Materials

Design of new microelectronic materials is characterized by several challenges such as high-dimensionality of the atomic structure-composition variable space, formidable cost of directly using high-fidelity simulations for design optimization, dispersity in literature-reported similar materials and synthesis methods, complex physical mechanisms, and mixed qualitative and quantitative design variables that lead to a disjointed design space. Even though machine learning (ML) techniques have been employed to expedite materials innovation, existing methods treat ML and design optimization as two separate processes, failing to resolve the fundamental challenges associated with high dimensionality and mixed-variable complexity. We have developed a ML enhanced mixed-variable material design optimization framework to efficiently extract useful information from existing data in literature and physics-based simulations to guide the autonomous search for optimal materials. Our proposed framework is composed of four computational modules: (1) a natural language processing (NLP) based virtual screening module, (2) classification based concept exploration module, (3) a density functional theory (DFT)-based high-fidelity evaluation model, and (4) a novel latent-variable Gaussian process (LVGP) ML model for mixed-variable problems with uncertainty quantification, which seamlessly integrates with Bayesian Optimization (BO) and achieves superb efficiency through embedded physics-based dimension reduction. Our approach is demonstrated and validated using the testbed of functional materials exhibiting metal-insulation transitions (MITs), with the targeted reversible resistivity changes (∼10^5) near room temperature. At the end of the 30-month project, we have developed a series of new ML techniques using NLP, conditional variational autoencoders, active learning, latent-variable Gaussian processes, integrated with Bayesian optimization. Our project has resulted in new predicted MITs compounds and improved understanding of MITs microscopic mechanisms, which in turn will revolutionize microelectronics science to provide energy-saving solutions. Our research has improved both creativity and efficiency in transforming rare-event discoveries of new functional materials to persistent innovations. In addition to open-sourcing the online MIT database and the classification model, the LVGP open source code has been downloaded more than 15,000 times within two years. More than 40 MIT compounds have been identified and many have been pursued experimentally via collaborators. The research results are published in close to 20 collaborative papers in high-impact journals, such as Chem. Mater., Appl. Phys. Rev., Sci. Rep., among others of design space.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Optimization of an aerostructural machining process using physics-guided Bayesian stability modelling

Existing algorithms for predicting milling chatter have not been widely adopted in industry since they require specialized instruments to measure the stability inputs. This study describes how the machining process for a meter-scale aluminum aerostructure was optimized using a physics-guided Bayesian stability model. The study was performed in collaboration with an industrial partner on production machines to evaluate the practicality of the proposed method under real-world conditions. For each cutting tool, the Bayesian approach automatically selected a small number of cutting tests, which were monitored using a microphone to observe the chatter frequency. The algorithm learned the system dynamics, cutting forces, and stability map from these test results. A novel algorithm for predicting tool bending stress was incorporated into the test selection algorithm to avoid tool breakage. On average, each set of optimized cutting parameters required less than six tests to identify and were 97% more productive than baseline parameters from the cutting tool manufacturer. The machining program was then further optimized using commercial feedrate scheduling software to remove cutting force spikes and reduce air cutting time. Five components were machined using the optimized process. These results demonstrate the potential for physics-guided Bayesian models to improve productivity in industrial settings.

Cornelius, Aaron [UT Knoxville]↗
DOE OSTI2024

Shock Hugoniot calculations using on-the-fly machine learned force fields with ab initio accuracy

We present a framework for computing the shock Hugoniot using on-the-fly machine learned force field (MLFF) molecular dynamics simulations. In particular, we employ an MLFF model based on the kernel method and Bayesian linear regression to compute the free energy, atomic forces, and pressure, in conjunction with a linear regression model between the internal and free energies to compute the internal energy, with all training data generated from Kohn–Sham density functional theory (DFT). We verify the accuracy of the formalism by comparing the Hugoniot for carbon with recent Kohn–Sham DFT results in the literature. In so doing, we demonstrate that Kohn–Sham calculations for the Hugoniot can be accelerated by up to two orders of magnitude, while retaining ab initio accuracy. We apply this framework to calculate the Hugoniots of 14 materials in the FPEOS database, comprising 9 single elements and 5 compounds, between temperatures of 10 kK and 2 MK. We find good agreement with first principles results in the literature while providing tighter error bars. In addition, we confirm that the inter-element interaction in compounds decreases with temperature.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2019

Model-based Hyperparameter Optimization of Convolutional Neural Networks for Information Extraction from Cancer Pathology Reports on HPC

Finding optimal hyperparameters is necessary to identify the best performing deep learning models but the process is costly. In this paper, we applied model-based optimization, also known as Bayesian optimization, using the CANDLE framework implemented on a High-Performance Computing environment. As a use case we selected information extraction from cancer pathology reports using a multi-task convolutional neural network, and hierarchical convolutional attention network to be optimized. We utilized a synthesized text corpus of 8,000 training cases and 2,000 validation cases with four types of clinical task labels including primary cancer site, laterality, behavior, and histological grade. We demonstrated that hyperparameter optimization using the CANDLE framework is a feasible approach with respect to both scalability and clinical task performance.

Yoon, Hong-Jun↗