Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Deep Neural Networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 775 records · Page 43

On the influence of over-parameterization in manifold based surrogates and deep neural operators

Constructing accurate and generalizable approximators (surrogate models) for complex physico-chemical processes exhibiting highly non-smooth dynamics is challenging. The main question is what type of surrogate models we should construct and should these models be under-parameterized or over-parameterized. In this work, we propose new developments and perform comparisons for two promising approaches: manifold-based polynomial chaos expansion (m-PCE) and the deep neural operator (DeepONet), and we examine the effect of over-parameterization on generalization. While m-PCE enables the construction of a mapping by first identifying low-dimensional embeddings of the input functions, parameters, and quantities of interest (QoIs), a neural operator learns the nonlinear mapping via the use of deep neural networks. Here, we demonstrate the performance of these methods in terms of generalization accuracy by solving the 2D time-dependent Brusselator reaction-diffusion system with uncertainty sources, modeling an autocatalytic chemical reaction between two species. We first propose an extension of the m-PCE by constructing a mapping between latent spaces formed by two separate embeddings of the input functions and the output QoIs. To further enhance the accuracy of the DeepONet, we introduce weight self-adaptivity in the loss function. We demonstrate that the performance of m-PCE and DeepONet is comparable for cases of relatively smooth input-output mappings. However, when highly non-smooth dynamics is considered, DeepONet shows higher approximation accuracy. We also find that for m-PCE, modest over-parameterization leads to better generalization, both within and outside of distribution, whereas aggressive over-parameterization leads to over-fitting. In contrast, an even highly over-parameterized DeepONet leads to better generalization for both smooth and non-smooth dynamics. Furthermore, we compare the performance of the above models with another recently proposed operator learning model, the Fourier Neural Operator, and show that its over-parameterization also leads to better generalization. Taken together, our studies show that m-PCE can provide very good accuracy at very low training cost, whereas a highly over-parameterized DeepONet can provide better accuracy and robustness to noise but at higher training cost. In both methods, the inference cost is negligible.

97 MATHEMATICS AND COMPUTING↗

Data-driven models of nonautonomous systems

Nonautonomous dynamical systems are characterized by time-dependent inputs, which complicates the discovery of predictive models describing the spatiotemporal evolution of the state variables of quantities of interest from their temporal snapshots. When dynamic mode decomposition (DMD) is used to infer a linear model, this difficulty manifests itself in the need to approximate the time-dependent Koopman operators. Our approach is to approximate the original nonautonomous system with a modified system derived via a local parameterization of the time-dependent inputs. The modified system comprises a sequence of local parametric systems, which are subsequently approximated by a parametric surrogate model using the DRIPS (dimension reduction and interpolation in parameter space) framework. The offline step of DRIPS relies on DMD to build a linear surrogate model, endowed with reduced-order bases for the observables mapped from training data. The online step interpolates on suitable manifolds to construct a sequence of iterative parametric surrogate models; the target/test parameter points on these manifolds are specified by a local parameterization of the test time-dependent inputs. Here, we use numerical experimentation to demonstrate the robustness of our method and compare its performance with that of deep neural networks.

97 MATHEMATICS AND COMPUTING↗

Learning macroscopic internal variables and history dependence from microscopic models

This paper concerns the study of history dependent phenomena in heterogeneous materials in a two-scale setting where the material is specified at a fine microscopic scale of heterogeneities that is much smaller than the coarse macroscopic scale of application. Here, we specifically study a polycrystalline medium where each grain is governed by crystal plasticity while the solid is subjected to macroscopic dynamic loads. The theory of homogenization allows us to solve the macroscale problem directly with a constitutive relation that is defined implicitly by the solution of the microscale problem. However, the homogenization leads to a highly complex history dependence at the macroscale, one that can be quite different from that at the microscale. In this paper, we examine the use of machine-learning, and especially deep neural networks, to harness data generated by repeatedly solving the finer scale model to: (i) gain insights into the history dependence and the macroscopic internal variables that govern the overall response; and (ii) to create a computationally efficient surrogate of its solution operator, that can directly be used at the coarser scale with no further modeling. We do so by introducing a recurrent neural operator (RNO), and show that: (i) the architecture and the learned internal variables can provide insight into the physics of the macroscopic problem; and (ii) that the RNO can provide multiscale, specifically FE 2 , accuracy at a cost comparable to a conventional empirical constitutive relation.

36 MATERIALS SCIENCE↗

Accelerating the discovery of battery electrode materials through data mining and deep learning models

The availability of crystalline materials databases allows for building accurate machine learning (ML) models that can accelerate the exploration of materials chemical space for energy storage applications. In this work, we screen all inorganic materials included in the Materials Project and AFLOW databases as potential metal-ion battery electrodes. We develop an efficient protocol to mine and screen raw data in current databases and provide a new database of electrode materials by considering pairs of charged and discharged electrodes. This effort leads to a new database with over 190,000 instances, in contrast to the original battery database which contains about 5000. The expanded battery data set is then used to build regression-based deep neural network models for predicting average voltages and percentage volume changes upon charging and discharging, which present improvements of at least 28% for target properties with respect to previous models, and are now able to predict anode electrodes (low voltage region) as well as electrodes that will not work in electrochemical cells (negative voltages), overcoming the challenges identified in previous ML models for battery electrodes. Additionally, a further screening of the expanded database itself allowed us to identify 35 novel electrode candidates with excellent battery performance metrics.

25 ENERGY STORAGE↗

A hybrid numerical and machine learning framework for evaluating the performance of a 780 cm 2 aqueous organic redox flow battery

Aqueous organic redox flow battery (AORFB) is a promising cost-competitive technology for large-scale energy storage. Among existing work, the dihydroxyphenazine (DHP)-based AORFB has demonstrated high energy density and low capacity degradation in 10 cm2 cells during lab tests. However, its commercial-scale performance in more complex environments remains unknown, posing a barrier for commercialization. To address this gap, this work presents a comprehensive performance evaluation of a 780 cm 2 DHP-based AORFB by combining physics-based numerical model, machine learning (ML)-based surrogate models, and ML-derived sensitivity quantification. Specifically, we first select 12 key battery parameters that include 10 physicochemical quantities and 2 operation quantities, then select 6 performance metrics that include energy efficiency (EE), discharging capacity, charging energy, and power losses due to concentration, activation, and ohmic over-potentials. With such selection, 12800 combinations of the 12 parameters are subsequently generated using the Latin Hypercube Sampling method. These combinations, together with 38 pre-defined State of Charge, are then integrated to a validated AORFB model developed in COMSOL to compute the performance metrics. With both input parameters and performance metrics, 60 deep neural network (DNN) surrogate models are then trained to approximate the relationship between the 10 physicochemical quantities and 6 performance metrics at each flow rate and current density. Sensitivity scores are then calculated based on the DNN models. Two additional sensitivity analysis tools, i.e., MARS, and SHAP, are also used to cross-validate the sensitivity scores from the DNN. The results demonstrate that 1) the standard potential ranks the first in controlling EE and charging energy, 2) the membrane conductivity is most critical for power loss and EE, and 3) specific area and reaction rate control activation power loss.

25 ENERGY STORAGE↗

An investigation on machine learning predictive accuracy improvement and uncertainty reduction using VAE-based data augmentation

The confluence of ultrafast computers with large memory, rapid progress in Machine Learning (ML) algorithms, and the availability of large datasets place multiple engineering fields at the threshold of dramatic progress. However, a unique challenge in nuclear engineering is data scarcity because experimentation on nuclear systems is usually more expensive and time-consuming than most other disciplines. One potential way to resolve the data scarcity issue is deep generative learning, which uses certain ML models to learn the underlying distribution of existing data and generate synthetic samples that resemble the real data. In this way, one can significantly expand the dataset to train more accurate predictive ML models. In this study, our objective is to evaluate the effectiveness of data augmentation using variational autoencoder (VAE)-based deep generative models. We investigated whether the data augmentation leads to improved accuracy in the predictions of a deep neural network (DNN) model trained using the augmented data. Additionally, the DNN prediction uncertainties are quantified using Bayesian Neural Networks (BNN) and conformal prediction (CP) to assess the impact on predictive uncertainty reduction. To test the proposed methodology, we used TRACE simulations of steady-state void fraction data based on the NUPEC Boiling Water Reactor Full-size Fine-mesh Bundle Test (BFBT) benchmark. Here, we found that augmenting the training dataset using VAEs has improved the DNN model’s predictive accuracy, improved the prediction confidence intervals, and reduced the prediction uncertainties.

Bayesian neural network↗

Search for W' bosons decaying to a top and a bottom quark at s=13TeV in the hadronic final state

A search is performed for W' bosons decaying to a top and a bottom quark in the all-hadronic final state, in proton-proton collisions at a center-of-mass energy of 13 TeV. The analyzed data were collected by the CMS experiment between 2016 and 2018 and correspond to an integrated luminosity of 137 fb -1 . Deep neural network algorithms are used to identify the jet initiated by the bottom quark and the jet containing the decay products of the top quark when the W' boson from the top quark decays hadronically. No excess above the estimated standard model background is observed. Upper limits on the production cross sections of W' bosons decaying to a top and a bottom quark are set. Both left- and right-handed W' bosons with masses below 3.4 TeV are excluded at 95% confidence level, and the most stringent limits to date on W bosons decaying to a top and a bottom quark in the all-hadronic final state are obtained.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Search for flavour-changing neutral-current couplings between the top quark and the photon with the ATLAS detector at s = 13 TeV

This letter documents a search for flavour-changing neutral currents (FCNCs), which are strongly suppressed in the Standard Model, in events with a photon and a top quark with the ATLAS detector. The analysis uses data collected in pp collisions at TeV during Run 2 of the LHC, corresponding to an integrated luminosity of 139 fb –1 . Both FCNC top-quark production and decay are considered. The final state consists of a charged lepton, missing transverse momentum, a b-tagged jet, one high-momentum photon and possibly additional jets. A multiclass deep neural network is used to classify events either as signal in one of the two categories, FCNC production or decay, or as background. No significant excess of events over the background prediction is observed and 95% CL upper limits are placed on the strength of left- and right-handed FCNC interactions. The 95% CL bounds on the branching fractions for the FCNC top-quark decays, estimated (expected) from both top-quark production and decay, are B(t → uγ) < 0.85 (0.88$^{+0.37}_{-0.25}$) × 10 –5 and B(t → cγ) < 4.2 (3.40$^{+1.35}_{-0.95}$) × 10 –5 for a left-handed tqγ coupling, and B(t → uγ) < 1.2 (1.20$^{+0.50}_{-1.33}$) ×10 –5 and B(t → cγ) < 4.5 (3.70$^{+1.47}_{-1.03}$) × 10 –5 for a right-handed coupling.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Measurement of the Higgs boson mass in the H → ZZ*→ 4ℓ decay channel using 139 fb –1 of s = 13 TeV pp collisions recorded by the ATLAS detector at the LHC

The mass of the Higgs boson is measured in the H→ZZ * →4ℓ decay channel. The analysis uses proton–proton collision data from the Large Hadron Collider at a centre-of-mass energy of 13 TeV recorded by the ATLAS detector between 2015 and 2018, corresponding to an integrated luminosity of 139 fb –1 . The measured value of the Higgs boson mass is 124.99 ± 0.18(stat.) ± 0.04(syst.) GeV. In final states with muons, this measurement benefits from an improved momentum-scale calibration relative to that adopted in previous publications. The measurement also employs an analytic model that takes into account the invariant-mass resolution of the four-lepton system on a per-event basis and the output of a deep neural network discriminating signal from background events. This measurement is combined with the corresponding measurement using 7 and 8 TeV pp collision data, resulting in a Higgs boson mass of 124.94 ± 0.17(stat.) ± 0.03(syst.) GeV.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Ion Distribution and Cation Exchange at Mica–Electrolyte Interfaces Probed with Deep Potential Molecular Dynamics

Here, we investigate the Stern layer structure and cation exchange mechanism at muscovite mica-electrolyte interfaces using nanosecond timescale molecular dynamics simulations based on deep neural network interatomic potentials trained on Density Functional Theory (DFT) data. Focusing on mica with exposed surface K + interfaced with aqueous NaCl and mica with surface Na + interfaced with KCl solution, we find that K + remains predominantly in inner-sphere configurations, while Na + exhibits notable populations in outer-sphere states. Most importantly, our simulations show that contact with an electrolyte solution results in the co-adsorption of multiple cation species, making the mica surface locally overcharged and thus reshaping the cation speciation in a manner that enhances the tendency of neighboring surface cations to desorb. These findings are consistent with recent experimental observations that co-adsorption of different cation species induces changes in cation speciation and slow kinetics of cation exchange at the muscovite-water interface, providing a basis for their detailed understanding.

36 MATERIALS SCIENCE↗

Introduction: Neuromorphic Materials

The explosive growth in data collection and the need to process it efficiently, as well as the desire to automate increasingly complex tasks in transportation, medical care, manufacturing, security and many other fields have motivated a growing interest in neuromorphic computing. Unlike the binary, transistorbased ON/OFF logic gates and separate logic and memory functionalities employed in digital computing, neuromorphic computing is inspired by animal brains that use interconnected synapses and neurons to perform processing, storage and transmission of information at the same location, while only consuming ~20 W or less of power. Motivated by the brain’s efficiency, adaptability, self-learning and resiliency qualities, neuromorphic computing can be broadly defined as an approach to processing and storing information using hardware and algorithms inspired by models of biological neural systems. Present research in neuromorphic computing encompasses approaches that vary significantly in their degree of neuro-inspiration, from systems that only incorporate features such as asynchronous, event-driven operation or use crossbar arrays of non-volatile memory (NVM) elements to accelerate deep neural networks (DNNs), to designs that embrace the extreme parallelism, sparsity, reconfigurability, adaptability, complexity and stochasticity observed in nervous systems. The term ‘neuromorphic’ computing is often credited to Carver Mead, who in the 1980s investigated Si-based analog electronics to replicate functions of the animal retina. Earlier important advances in this field include the work of Frank Rosenblatt, who proposed the concept of the perceptron, Bernard Widrow, who used this concept to build one of the first analog neural networks, the Adaline and many other researchers (see ref. 6 for an historical perspective on neuromorphic computing). With the recent increase in the use of artificial intelligence and large language models, and rising concerns over the associated energy costs, interest in neuromorphic hardware has expanded rapidly. According to some estimates, driven largely by the drastic growth in the training use of artificial intelligence (AI) models using the current computing architectures, the energy cost of computing is projected to reach the energy supply worldwide by 2045. Furthermore, while this is not a realistic outcome, it means that, if more efficient computing technologies are not developed -- soon -- the world will soon become one where demand for energy and market constraints limit the continued increase of societal access to AI and cloud services from data centers. Data centers used for training and use of these models consume hundreds of terawatt hours of electricity, already past 4% of the US electricity demand.

Circuits↗

Phase Equilibrium of Water with Hexagonal and Cubic Ice Using the SCAN Functional

Machine learning models are rapidly becoming widely used to simulate complex physicochemical phenomena with ab initio accuracy. Here, we use one such model as well as direct density functional theory (DFT) calculations to investigate the phase equilibrium of water, hexagonal ice (Ih), and cubic ice (Ic), with an eye toward studying ice nucleation. The machine learning model is based on deep neural networks and has been trained on DFT data obtained using the SCAN exchange and correlation functional. We use this model to drive enhanced sampling simulations aimed at calculating a number of complex properties that are out of reach of DFT-driven simulations and then employ an appropriate reweighting procedure to compute the corresponding properties for the SCAN functional. This approach allows us to calculate the melting temperature of both ice polymorphs, the driving force for nucleation, the heat of fusion, the densities at the melting temperature, the relative stability of ices Ih and Ic, and other properties. We find a correct qualitative prediction of all properties of interest. In some cases, quantitative agreement with experiment is better than for state-of-the-art semiempirical potentials for water. Furthermore, our results also show that SCAN correctly predicts that ice Ih is more stable than ice Ic.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Efficient Mixed-Precision Matrix Factorization of the Inverse Overlap Matrix in Electronic Structure Calculations with AI-Hardware and GPUs

In recent years, a new kind of accelerated hardware has gained popularity in the artificial intelligence (AI) community which enables extremely high-performance tensor contractions in reduced precision for deep neural network calculations. In this article, we exploit Nvidia Tensor cores, a prototypical example of such AI-hardware, to develop a mixed precision approach for computing a dense matrix factorization of the inverse overlap matrix in electronic structure theory, S –1 . This factorization of S –1 , written as ZZT = S –1 , is used to transform the general matrix eigenvalue problem into a standard matrix eigenvalue problem. Here we present a mixed precision iterative refinement algorithm where Z is given recursively using matrix–matrix multiplications and can be computed with high performance on Tensor cores. To understand the performance and accuracy of Tensor cores, comparisons are made to GPU-only implementations in single and double precision. Additionally, we propose a nonparametric stopping criteria which is robust in the face of lower precision floating point operations. The algorithm is particularly useful when we have a good initial guess to Z, for example, from previous time steps in quantum-mechanical molecular dynamics simulations or from a previous iteration in a geometry optimization.

36 MATERIALS SCIENCE↗

PySIDT: Subgraph Isomorphic Decision Trees for Molecular Property Prediction

Accurate molecular property prediction is important across all fields of chemistry. Deep neural networks (DNNs) have become increasingly popular due to their ability to train automatically, avoiding the incredibly tedious process of constructing and extending traditional property estimation schemes. However, DNNs require large amounts of training data, are challenging to interpret, require large amounts of memory to load even during inference, and have severe difficulties incorporating qualitative chemical knowledge, which are often desired for molecular property prediction tasks. Here, in this study, we present PySIDT (https://github.com/zadorlab/PySIDT), a software for training and running inference on Subgraph Isomorphic Decision Trees (SIDTs). SIDTs are graph-based decision trees made of nodes associated with molecular substructures. Inference is done by descending target molecular structures down the decision tree to nodes with matching subgraph isomorphic substructures and making predictions based on the final (most specific) nodes matched. SIDTs scale down well to dataset sizes much smaller than is feasible for DNNs. As trees of molecular substructures, SIDTs are inherently readable and easy to visualize, making them easy to analyze. They are also straightforward to extend and retrain, facilitate uncertainty estimation, and enable easy integration of expert knowledge. We demonstrate the SIDT approach discussing its application to a diverse range of molecular prediction tasks: rate coefficient estimation, diffusion coefficient estimation, thermochemistry estimation, transition state bond stretch prediction, p K a prediction, stability of molecular structures, stability of surface structures, and prediction of surface lateral interaction energetics. Additionally, we demonstrate the power of the SIDT algorithms in two direct learning curve vanilla comparisons with the popular DNN-based software Chemprop and the popular gradient boosted trees-based software XGBoost on enthalpy of formation and rate coefficient prediction tasks. In particular, in the enthalpy of formation case, vanilla PySIDT is able to outperform vanilla Chemprop and XGBoost across the full range of training/validation set sizes out to 11,560 data points.

Johnson, Matthew Sean [Sandia National Laboratorie↗

Adaptive Data-Driven Deep-Learning Surrogate Model for Frontal Polymerization in Dicyclopentadiene

Frontal polymerization (FP) is a self-sustaining curing process that enables rapid and energy-efficient manufacturing of thermoset polymers and composites. Computational methods conventionally used to simulate the FP process are time-consuming, and repeating simulations are required for sensitivity analysis, uncertainty quantification, or optimization of the manufacturing process. Here, in this work, we develop an adaptive surrogate deep-learning model for FP of dicyclopentadiene (DCPD), which predicts the evolution of temperature and degree of cure orders of magnitude faster than the finite-element method (FEM). The adaptive algorithm provides a strategy to select training samples efficiently and save computational costs by reducing the redundancy of FEM-based training samples. The adaptive algorithm calculates the residual error of the FP governing equations using automatic differentiation of the deep neural network. A probability density function expressed in terms of the residual error is used to select training samples from the Sobol sequence space. The temperature and degree of cure evolution of each training sample are obtained by a 2D FEM simulation. The adaptive method is more efficient and has a better prediction accuracy than the random sampling method. With the well-trained surrogate neural network, the FP characteristics (front speed, shape, and temperature) can be extracted quickly from the predicted temperature and degree-of-cure fields.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Modeling Nanoconfinement Effects Using Active Learning

Predicting the spatial configuration of gas in nanopores of is relevant in applications such as fluid flow forecasting and hydrocarbon reserves estimation. For example, shale reservoirs have suffered from computationally intractable multiscale problems, since fluid properties such as viscosity, density, and adsorption must be calculated by using expensive molecular dynamics (MD) simulations within each nanopore, whereas flow through these connected nanopores must be simulated at the micrometer scale. We utilize machine learning techniques to quickly and accurately model nanoscale confinement effects as an important step toward bridging the nano and micro scales. Our workflow is based on building and training physics-based deep-neural-networks models by learning from a database of MD calculations. The model accounts for the adsorption phenomenon by predicting the statistical distribution of gas inside nanopores. Because large databases of MD calculations are expensive to create, we investigate active learning (AL) as a data set construction strategy. In this workflow, new data are selected based on the model uncertainty via the query-by-committee approach. We show that our workflow obtains accurate models that generalize to real scanning electron microscopy geometries with 1/10th of the number of MD calculations required vs random data set generation. Our method enables the possibility of modeling nanoconfinement effects at the mesoscale, where complex connected sets of nanopores affect flow.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Machine Learning Automated Analysis of Enormous Synchrotron X-ray Diffraction Datasets

X-ray diffraction (XRD) data analysis can be a time-consuming and laborious task. Deep neural network (DNN) based models trained with synthetic XRD patterns have been proven to be a highly efficient, accurate, and automated method for analyzing common XRD data collected from solid samples in ambient environments. However, it remains unclear whether synthetic XRD-based models can be effective in solving micro(μ)-XRD mapping data for in situ experiments involving liquid phases, which always have lower quality and significant artifacts. In this study, we collected μ-XRD mapping data from a LaCl 3 -calcite hydrothermal fluid system and trained two categories of models to analyze the experimental XRD patterns. Here, the models trained solely with synthetic XRD patterns showed low accuracy (as low as 64%) when solving experimental μ-XRD mapping data. However, the accuracy of the DNN models significantly improved (90% or above) when we trained them with a data set containing both synthetic and a small number of labeled experimental μ-XRD patterns. This study highlights the importance of labeled experimental patterns in training DNN models to solve μ-XRD mapping data from in situ experiments involving liquid phases.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Machine Learning the Voltage of Electrode Materials in Metal-Ion Batteries

Machine-learning (ML) techniques have rapidly found applications in many domains of materials chemistry and physics where large data sets are available. Aiming to accelerate the discovery of materials for battery applications, in this work, we develop a tool based on ML models to predict voltages of electrode materials for metal-ion batteries. To this end, we use deep neural network, support vector machine, and kernel ridge regression as ML algorithms in combination with data taken from the Materials Project database, as well as feature vectors from properties of chemical compounds and elemental properties of their constituents. We show that our ML models have predictive capabilities for different reference test sets and, as an example, we utilize them to generate a voltage profile diagram and compare it to density functional theory calculations. In addition, using our models, we propose nearly 5000 candidate electrode materials for Na- and K-ion batteries. We also make available a web-accessible tool that, within a minute, can be used to estimate the voltage of any bulk electrode material for a number of metal ions. Here, these results show that ML is a promising alternative for computationally demanding calculations as a first screening tool of novel materials for battery applications.

25 ENERGY STORAGE↗