Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “neural generative models”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11

Computational Workflow for Accelerated Molecular Design Using Quantum Chemical Simulations and Deep Learning Models

Efficient methods for searching the chemical space of molecular compounds are needed to automate and accelerate the design of new functional molecules such as pharmaceuticals. Given the high cost in both resources and time for experimental efforts, computational approaches play a key role in guiding the selection of promising molecules for further investigation. Here, we construct a workflow to accelerate design by combining approximate quantum chemical methods [i.e. density-functional tight-binding (DFTB)], a graph convolutional neural network (GCNN) surrogate model for chemical property prediction, and a masked language model (MLM) for molecule generation. Property data from the DFTB calculations are used to train the surrogate model; the surrogate model is used to score candidates generated by the MLM. The surrogate reduces computation time by orders of magnitude compared to the DFTB calculations, enabling an increased search of chemical space. Furthermore, the MLM generates a diverse set of chemical modifications based on pre-training from a large compound library. We utilize the workflow to search for near-infrared photoactive molecules by minimizing the predicted HOMO-LUMO gap as the target property. Our results show that the workflow can generate optimized molecules outside of the original training set, which suggests that iterations of the workflow could be useful for searching vast chemical spaces in a wide range of design problems.

Blanchard, Andrew↗

Co-optimized machine-learned manifold models for large eddy simulation of turbulent combustion

Many modeling approaches in large eddy simulation (LES) of turbulent combustion employ a projection of the thermochemical state onto a low-dimensional manifold within state space to reduce the number of transported variables and hence computational cost. Flamelet-generated manifolds (FGM) is an example of a well-established, physics-based approach, but increasingly, principal component analysis (PCA) is being used as a data-driven method for generating manifold models. For both approaches, the nonlinear relationship between the location on the predefined manifold and the outputs of interest, such as reaction rates, can be tabulated or encoded in a neural network. This work proposes a new approach for manifold modeling that extends these existing approaches. A modified neural network structure simultaneously encodes the definition of the manifold variables, the nonlinear mapping, and the subfilter closure for LES. This allows all three of these aspects of the model to be co-optimized, generating a model from any source of combustion thermochemical state data. The manifold parameterizing variables are constrained to be linear combinations of species, as in FGM and PCA-based models, to aid in interpretability and implementation. For LES, subfilter variances of the manifold variables are also included as inputs. Two types of a priori analysis are performed to evaluate the new approach. In the first, the model is trained on data from one-dimensional premixed flames. In this case, the approach recovers the behavior of flamelet-based manifold approaches, and in fact slightly improves performance by identifying an optimized progress variable. The approach is also applied to data from direct numerical simulations of spherical ignition kernels in isotropic turbulence. For any specified manifold dimensionality, the new approach provides substantially lower prediction errors than a PCA-based model developed from the same data set. Additionally, the LES formulation of the new approach can provide accurate predictions for filtered reaction rates across a variety of filter widths.

97 MATHEMATICS AND COMPUTING↗

Fitting a deep generative hadronization model

Hadronization is a critical step in the simulation of high-energy particle and nuclear physics experiments. As there is no first principles understanding of this process, physically-inspired hadronization models have a large number of parameters that are fit to data. Deep generative models are a natural replacement for classical techniques, since they are more flexible and may be able to improve the overall precision. Proof of principle studies have shown how to use neural networks to emulate specific hadronization when trained using the inputs and outputs of classical methods. However, these approaches will not work with data, where we do not have a matching between observed hadrons and partons. In this paper, we develop a protocol for fitting a deep generative hadronization model in a realistic setting, where we only have access to a set of hadrons in data. Our approach uses a variation of a Generative Adversarial Network with a permutation invariant discriminator. We find that this setup is able to match the hadronization model in Herwig with multiple sets of parameters. This work represents a significant step forward in a longer term program to develop, train, and integrate machine learning-based hadronization models into parton shower Monte Carlo programs.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Development of a hybrid neural network and transfer learning model for optimized ICP-MS/MS operation

Correct function and calibration of instrumentation is a crucial assumption for any scientific experiment. One such instrument, tandem inductively coupled plasma mass spectrometer (ICP-MS/MS), has in-depth calibration settings that range across 30+ different parameters, making it difficult to determine optimal conditions without expertise and some degree of trial and error. Often, these settings are hand-tuned, a time-intensive process prone to local maxima and human error. While some automation is available, the automation also may favor local optimizations over a global optimum. In addition to these difficulties, day to day instrument variability can further complicate the calibration process. We propose a solution to this problem as a machine learning (ML) algorithm that learns how each parameter helps determine the calibration sensitivity across several elements, and re-weights parameters over time as instrument variability changes (e.g., a global neural network (NN) with a time-dependent transfer learning (TL) component). This model would be able to generate a surface of predicted calibration sensitivities and their respective parameters, and a simple multivariate algorithm would be able to pull out the optimum results with the settings associated with them. Here-in, we describe our initial findings in working towards this goal, including data extraction from historical files, exploratory data analysis, and some initial model building to better describe the data and the feasibility of our goal.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Probabilistic Prediction of Geomagnetic Storms and the K p index

Geomagnetic activity is often described using summary indices to summarize the likelihood of space weather impacts, as well as when parameterizing space weather models. The geomagnetic index K p in particular, is widely used for these purposes. Current state-of-the-art forecast models provide deterministic K p predictions using a variety of methods – including empirically-derived functions, physics-based models, and neural networks – but do not provide uncertainty estimates associated with the forecast. This paper provides a sample methodology to generate a 3-hour-ahead K p prediction with uncertainty bounds and from this provide a probabilistic geomagnetic storm forecast. Specifically, we have used a two-layered architecture to separately predict storm (K p ≥ 5 – ) and non-storm cases. As solar wind-driven models are limited in their ability to predict the onset of transient-driven activity we also introduce a model variant using solar X-ray flux to assess whether simple models including proxies for solar activity can improve the predictions of geomagnetic storm activity with lead times longer than the L1-to-Earth propagation time. By comparing the performance of these models we show that including operationally-available information about solar irradiance enhances the ability of predictive models to capture the onset of geomagnetic storms and that this can be achieved while also enabling probabilistic forecasts.

79 ASTRONOMY AND ASTROPHYSICS↗

An adaptive adversarial domain adaptation approach for corn yield prediction

Recently, statistical machine learning and deep learning methods have been widely explored for corn yield prediction. Though successful, machine learning models generated within a specific spatial domain often lose their validity when directly applied to new regions. To address this issue, we designed an unsupervised adaptive domain adversarial neural network (ADANN). Specifically, through domain adversarial training, the ADANN model reduced the impact of domain shift by projecting data from different domains into the same subspace. Also, the ADANN model was designed to be trained in an adaptive way, which guaranteed the model can learn the domain-invariant features and perform accurate yield prediction simultaneously. Informative variables including time-series vegetation indices and sequential weather observations were first collected from multiple data sources and aggregated to the county level. Then, we trained the ADANN model with the extracted features and corresponding reported county-level corn yield from the U.S. Department of Agriculture (USDA). Finally, the trained model was evaluated in four testing years 2016–2019. The U.S. corn belt was used as the study area and counties under study were grouped into two diverse ecological regions. Overall, the experimental results showed that the developed ADANN model had better performance than three other state-of-the-art machine learning models in both local experiments (train and test in the same region) and transfer experiments (train and test in different regions). As the first study using adversarial learning for crop yield prediction, this research demonstrates a novel solution for improving model transferability on crop yield prediction.

59 BASIC BIOLOGICAL SCIENCES↗

Exploring 2D X-ray diffraction phase fraction analysis with convolutional neural networks: Insights from kinematic-diffraction simulations

Abstract Deep-learning models are effective for analyzing the complex information in 2D X-ray diffraction (XRD) patterns. Accurately collecting parameters of the material sample is crucial during model training, significantly impacting model performance. In this study, we employ a kinematic-diffraction simulator to generate simulated 2D XRD patterns for Ti–6Al–4V alloy, allowing precise control of sample parameters. These simulated patterns are used to train convolutional neural networks, predicting $$\upbeta$$ β -phase volume fractions. The training data set consists exclusively of 2D XRD patterns with pure $$\upalpha$$ α - or pure $$\upbeta$$ β -phase, while the testing set incorporates patterns with intermediate phase volume fraction. In particular, we investigate how the architectures of the model influence prediction reliability and computational performance. Experimental results reveal that, with appropriate training, the convolutional neural network accurately detects intermediate phase volume fractions even trained with only pure-phase patterns, achieving a mean square error accuracy of $$9.4 \times 10^{-4}$$ 9.4 × 10 - 4 . Graphical abstract

Yue, Weiqi↗

Machine learning elastic constants of multi-component alloys

The present manuscript explores application of machine learning methods for determining elastic constants and other derived mechanical properties of multi-component alloys. Here, a number of machine learning models, including linear regression, neural network and random forest based models, are trained and tested on a dataset of binary alloys generated using density functional theory (DFT) calculations and spanning over a large number of elemental species in the periodic table. Starting with a wide range of simple and easily accessible compositionally-averaged elemental features, a correlation-based feature selection strategy was used to systematically down-select a set of most relevant features towards the prediction of the elasticity tensor components. The true predictive performance and the associated uncertainties of the models were established by testing on unseen data and bootstrapping, respectively. A single and pair-wise feature partial dependence analysis was performed to visualize the average property trends in the multi-dimensional feature space in order to further understand the achieved predictive performance. The utility of the trained model is further demonstrated by obtaining sufficiently accurate yet highly efficient approximations for bulk modulus, Young’s modulus, shear modulus and Poisson’s ratio for alloys beyond the binary space (i.e., two-component alloys) on which the model was originally trained. More importantly, we test and validate the predictive performance of the developed model directly against the experimentally measured elastic constants of technologically relevant multi-component alloys (such as, Ni- and Ti-based alloys). Finally, utility of such a data-enabled route is demonstrated by predicting the possible range of various elastic properties for vast composition space available within the five component Ni-Cr-Fe-Mo-W alloy system in a high-throughput manner.

36 MATERIALS SCIENCE↗

Discriminative versus generative approaches to simulation-based inference

Most of the fundamental, emergent, and phenomenological parameters of particle and nuclear physics are determined through parametric template fits. Simulations are used to populate histograms which are then matched to data. This approach is inherently lossy, since histograms are binned and low-dimensional. Deep learning has enabled unbinned and high-dimensional parameter estimation through neural likelihood(-ratio) estimation. We compare two approaches for neural simulation-based inference (NSBI): one based on discriminative learning (classification) and one based on generative modeling. These two approaches are directly evaluated on the same datasets, with a similar level of hyperparameter optimization in both cases. In addition to a Gaussian dataset, we study NSBI using a Higgs boson dataset from the FAIR Universe Challenge. We find that both the direct likelihood and likelihood ratio estimation are able to effectively extract parameters with reasonable uncertainties. For the numerical examples and within the set of hyperparameters studied, we found that the likelihood ratio method is more accurate and/or precise. Both methods have a significant spread from the network training and would require ensembling or other mitigation strategies in practice.

high energy physics↗

SOC Microstructural Property Estimator

This pre-trained ML model is a tool that uses basic compositional parameters for porous solid oxide cell (SOC) electrodes - the phase fractions and mean particle/pore diameters – as inputs and uses them to estimate additional electrochemical performance parameters: active (i.e., connected) TPB density, all tortuosity factors, and phase pair specific interfacial areas. The electrode is assumed to be composed of two solid phases and a pore phase. The property calculations are performed using neural network regression models trained on a large bank of synthetic electrode microstructural data that NETL has generated using the program DREAM3D (that bank is also hosted on EDX: https://edx.netl.doe.gov/dataset/soc-synthetic-microstructure-bank). This means the generated parameters are based on training from actual measured properties from 3D microstructures, not estimated from geometric simplifications. This tool was developed and is intended to replace percolation theory calculations in models that use hypothetical electrode properties. An example use case would be running SOC performance simulations across a parametric sweep of electrode designs (e.g., varying phase fractions and particle sizes) and assessing how it impacts the electrochemical performance of the SOC. Within the parameter space of the training data (statistics of that parameter space is provided in the readme file), this model achieves sub-5% mean absolute percent errors, an order of magnitude less error than percolation theory across the same parameter space. However, be aware that this tool was developed with parametric simulations in mind, and users are encouraged to assess accuracy for their own specific use case rather than taking accuracy metrics at face value. More info, including a usage guide, is in the included readme file. This tool should be cited with the DOI number provided.

Electrode Microstructure↗

AI-Improved Resolution Projections of Population Characteristics and Imperviousness Can Improve Resolution and Accuracy of Urban Flood Predictions

Focal Area: (2) Predictive modeling through the use of AI techniques and AI-derived model components. Specifically, we call for the use of deep generative models from ML for providing high-resolution projections of impervious surface area, as well as neural network solvers for fast approximation of urban hydrodynamics to provide greatly enhanced forecasts of future urban flood dynamics.

54 ENVIRONMENTAL SCIENCES↗

FY24 Progress Report: SRNL Analysis of ICCWR LCM and WAMS data for Corrosion and Cracking

Algorithms for Machine Learning (ML) and data analysis for the 3013 Surveillance Program have been developed in an ongoing collaborative effort by the Savannah River National Laboratory (SRNL) and the University of South Carolina (USC). The objective of the algorithms is to automate the identification of corrosion and crack formation in the Inner Container Closure Weld Region (ICCWR) of the canister system used to store Pu-bearing material. Data for corrosion and cracking is collected from large binary files generated by a Laser Confocal Microscope (LCM), the Wide Area 3D Measurement System (WAMS), or,in a recent proposal, by a Scanning Electron Microscope (SEM). The ML software uses the physical attributes in the data files (e.g., one or more of: height, color, and 16-bit grayscale values as functions of position in a plane projection) to detect signs of surface corrosion and cracking after being trained on similar data, with the features to be detected. Although the initial scope included screening for broader indicators of corrosion, e.g., pitting, identification of potential cracks was prioritized for the past several years at the request of program leadership. Labeled training data is essential to developing the ML algorithm, and enhancements to data labeling capability have been developed to address this essential precursor to application of ML routines. Efficient labeling is particularly important in view of the large volume of data required to train ML algorithms and the relative rarity of cracks in the ICCWR data set. The updated program will read binary data from either LCM, WAMS or SEM files, interrogate data attributes, facilitate user labeling of data for training ML algorithms, execute ML algorithms, output parameters from trained ML algorithms, report ML model accuracy with respect to labeled data, and generate graphical representations for various analyses. In FY24, hourglass neural networks (HNNs) that were initiated in FY22 were further developed and tested using available LCM data, and their performance was tested against that of the alternative U-Net Neural Network algorithm structure. HNNs along with previously developed Convolutional Neural Networks (CNNs) and Deep Neural Networks (DNNs) comprise a suite of ML tools for identification of cracks in the ICCWR

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Gaussian Process Regression for Aggregate Baseline Load Forecasting

Demand response (DR) is one of the most effective ways to maintain the reliability and improve the flexibility of power systems. Accurate forecasts of baseline loads are essential for DR programs. In the era of big data, machine learning-based approaches present a unique opportunity for baseline load forecasting. Thus, this paper presents a machine learning-based approach using a relatively less explored algorithm, Gaussian process regression (GPR), to forecast aggregate baseline loads. As such, a dataset was generated using a set of EnergyPlus simulations. Using the generated dataset, a GPR-based forecasting model was developed. In addition, support vector regression (SVR)-, artificial neural network (ANN)-, and averaging-based models were developed as baseline models for comparison. These models were compared in terms of accuracy, simplicity, and integrity. The prediction performance of the models showed that the GPR-based model is more accurate and reliable than the others. Such high performance shows the potential of the GPR in baseline load forecasting. GPR, therefore, can be used for DR applications.

Amasyali, Kadir↗

Improving Robustness of Spectrogram Classifiers with Neural Stochastic Differential Equations

Signal analysis and classification is fraught with high levels of noise and perturbation. Computer-vision-based deep learning models applied to spectrograms have proven useful in the field of signal classification and detection; however, these methods aren't designed to handle the low signal-to-noise ratios inherent within non-vision signal processing tasks. While they are powerful, they are currently not the method of choice in the inherently noisy and dynamic critical infrastructure domain, such as smart-grid sensing, anomaly detection, and non-intrusive load monitoring. Currently, these models can be brittle, which makes them susceptible to noisy input. This also means they have sub-optimal stability of explanation outputs. Experts and technicians using these models to make decisions in real world scenarios need assurance that a model is performing as it is supposed to. The classification or prediction outputs it generates should be sound and grounded, not likely to change in the presence of shifting noise landscapes. In this work, we explore the idea of Neural Stochastic Differential Equations (NSDE's) to improve the robustness of models trained to classify time series data and the effect of NSDE's on the explainability of outputs. We then test the effectiveness of these approaches by applying them to a non-intrusive load monitoring (NILM) dataset that consists of simulated harmonic signals injected into a real building.

Brogan, Joel↗

Prediction of laser beam spatial profiles in a high-energy laser facility by use of deep learning

We adapt the significant advances achieved recently in the field of generative artificial intelligence/machine-learning to laser performance modeling in multipass, high-energy laser systems with application to high-shot-rate facilities relevant to inertial fusion energy. Advantages of neural-network architectures include rapid prediction capability, data-driven processing, and the possibility to implement such architectures within future low-latency, low-power consumption photonic networks. Four models were investigated that differed in their generator loss functions and utilized the U-Net encoder/decoder architecture with either a reconstruction loss alone or combined with an adversarial network loss. We achieved inference times of 1.3 ms for a 256 × 256 pixel near-field beam with errors in predicted energy of the order of 1% over most of the energy range. It is shown that prediction errors are significantly reduced by ensemble averaging the models with different weight initializations. These results suggest that including the temporal dimension in such models may provide accurate, real-time spatiotemporal predictions of laser performance in high-shot-rate laser systems.

47 OTHER INSTRUMENTATION↗

Hypothesis testing via AI: Generating physically interpretable models of scientific data with machine learning (Full Technical Report)

Deep learning has demonstrated an exceptional ability to solve complex tasks (an engineering success); however, it has done so at the expense of the ability to generate new knowledge (a scientific failure). We propose an alternative framework—entitled Deep Symbolic Regression (DSR)—in which artificial neural networks (NNs) rapidly generate hypotheses about physical relationships among inputs. This framework bypasses the need to interpret an NN altogether, while still leveraging the representational power of deep learning. The resulting models are tractable mathematical expressions, which are inherently and readily human interpretable and can provide insights into underlying physical phenomena. Further, we fold this methodology into the scientific process by allowing the scientist to directly integrate a priori knowledge and beliefs to accelerate learning. We demonstrate this methodology on symbolic regression—the problem of rediscovering underlying expressions describing a dataset—and achieve state-of-the-art performance across a wide variety of symbolic regression problems. Further, we generalize our DSR framework to apply to the more general class of symbolic optimization problems, in which one seeks to optimize a sequence of symbols or “tokens” under a black-box reward function. Examples of other symbolic optimization problems include neural architecture search and computational antibody design. Our generalized tool, Deep Symbolic Optimization (DSO), has been demonstrated on the task of learning symbolic control policies for reinforcement learning environments, and has been adopted as an enabling capability for computational antibody design.

97 MATHEMATICS AND COMPUTING↗

A deep generative model for deciphering cellular dynamics and in silico drug discovery in complex diseases

Human diseases are characterized by intricate cellular dynamics. Single-cell transcriptomics provides critical insights, yet a persistent gap remains in computational tools for detailed disease progression analysis and targeted in silico drug interventions. Here we introduce UNAGI, a deep generative neural network tailored to analyse time-series single-cell transcriptomic data. This tool captures the complex cellular dynamics underlying disease progression, enhancing drug perturbation modelling and screening. When applied to a dataset from patients with idiopathic pulmonary fibrosis, UNAGI learns disease-informed cell embeddings that sharpen our understanding of disease progression, leading to the identification of potential therapeutic drug candidates. Validation using proteomics reveals the accuracy of UNAGI’s cellular dynamics analysis, and the use of the fibrotic cocktail-treated human precision-cut lung slices confirms UNAGI’s predictions that nifedipine, an antihypertensive drug, may have anti-fibrotic effects on human tissues. UNAGI’s versatility extends to other diseases, including COVID, demonstrating adaptability and confirming its broader applicability in decoding complex cellular dynamics beyond idiopathic pulmonary fibrosis, amplifying its use in the quest for therapeutic solutions across diverse pathological landscapes.

Neural Network↗

Big Sky Regional Carbon Sequestration Partnership (Phase III Final Scientific/Technical Report)

The Big Sky Carbon Sequestration Partnership (BSCSP) pursued a Phase III demonstration project at Kevin Dome in north central Montana. Kevin Dome covers approximately 700 square miles and is a naturally occurring CO 2 reservoir that is flanked by oil and gas fields. The carbon dioxide (CO 2 ) is in the upper Devonian Duperow (carbonate) formation and does not reach the spill point of the dome; therefore, the dome has potential as a CO 2 sequestration reservoir, a CO 2 supply, or as both if anthropogenic sources and enhanced oil recovery (EOR) operations are associated with the dome. Kevin Dome could potentially act as a buffer to continue accepting anthropogenic CO 2 when EOR flooding operations are interrupted or completed. The project objective was to produce one million tonnes of CO 2 from the gas cap of Kevin Dome, pipe it laterally, inject, and re-store it in the brine leg of the same formation to test the hub / buffer storage concept. This was to be accomplished by drilling up to five production wells, building a short pipeline and compression facilities, and drilling an injection well and several monitoring wells. BSCSP commenced outreach and site characterization activities including acquiring baseline data for near-surface insurance monitoring, acquiring 3-dimensional, 9-component surface seismic over the project area, drilling two test wells (one in the production area and one in the injection area), coring key intervals, and performing comprehensive logging. Well tests of those wells revealed two barriers to the project. The production (Danielson 33-17) well was perforated in multiple zones but failed to produce any significant CO 2 . This was despite being drilled in the near vicinity of a historic well that had produced 3700 MCF per day in a drill stem test. Modeling indicated that this was likely due to a phase change during production causing a temperature drop resulting in hydrate and/or water ice formation that clogged the formation. Tests of the injection zone (Wallewein 22-1) well indicated total dissolved solids (TDS) slightly below the EPA required 10,000 parts per million (ppm) for a Class VI underground injection control permit. While the project was initiated before Class VI rules were promulgated, and this was an experimental project (seemingly qualified for a Class V permit), the Environmental Protection Agency (EPA) indicated that injection would require a Class VI permit. The low salinity result was unexpected as contours plotted based on regional formation water quality data indicated an expected TDS above 20,000 ppm, and wells between the recharge zone and the Wallewein well tested above 10,000 ppm. Faced with the inability to obtain an injection permit, the demonstration project could not proceed. However, the project had generated valuable samples and data on a large natural analog including 32 sq. mi. of 3-D, 9-C seismic, 430 ft. of carbonate core covering seven different depositional environments taken from areas with, and without the presence of CO 2 , 30 ft. of core of two caprocks, a tight carbonate and an anhydrite, a full set of modern logs on both wells, and well tests. DOE decided to re-scope the project around completing studies utilizing this data. This report covers both the initial scope and the re-scope (Task / Section numbers preceded with an R). While the report covers a wide range of project activities, highlights of this work include: Development of a geostatic model using neural nets to match well logs to facies and using multi-waveform seismic to inform reservoir heterogeneity; Unique mechanical testing of permeability – stress relationship in two caprock materials; Development of full waveform inversion to generate a high resolution velocity model; Model development for dual permeability (fracture and matrix) systems to better account for matrix-matrix interactions; Joint seismic wave inversion (including the first quadr-joint inversion) exhibiting better imaging of a challenging reservoir zone in stiff rock; Core flow and core flood results on a reactive carbonate; and Innovative laboratory measurements of seismic response of fractured core as a function of fluid fill.

54 ENVIRONMENTAL SCIENCES↗