Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Deep Neural Networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36

Scalable algorithms for physics-informed neural and graph networks

Physics-informed machine learning (PIML) has emerged as a promising new approach for simulating complex physical and biological systems that are governed by complex multiscale processes for which some data are also available. In some instances, the objective is to discover part of the hidden physics from the available data, and PIML has been shown to be particularly effective for such problems for which conventional methods may fail. Unlike commercial machine learning where training of deep neural networks requires big data, in PIML big data are not available. Instead, we can train such networks from additional information obtained by employing the physical laws and evaluating them at random points in the space–time domain. Such PIML integrates multimodality and multifidelity data with mathematical models, and implements them using neural networks or graph networks. Here, we review some of the prevailing trends in embedding physics into machine learning, using physics-informed neural networks (PINNs) based primarily on feed-forward neural networks and automatic differentiation. For more complex systems or systems of systems and unstructured data, graph neural networks (GNNs) present some distinct advantages, and here we review how physics-informed learning can be accomplished with GNNs based on graph exterior calculus to construct differential operators; we refer to these architectures as physics-informed graph networks (PIGNs). We present representative examples for both forward and inverse problems and discuss what advances are needed to scale up PINNs, PIGNs and more broadly GNNs for large-scale engineering problems.

42 ENGINEERING↗

Data-driven discovery of coordinates and governing equations

The discovery of governing equations from scientific data has the potential to transform data-rich fields that lack well-characterized quantitative descriptions. Advances in sparse regression are currently enabling the tractable identification of both the structure and parameters of a nonlinear dynamical system from data. The resulting models have the fewest terms necessary to describe the dynamics, balancing model complexity with descriptive ability, and thus promoting interpretability and generalizability. This provides an algorithmic approach to Occam’s razor for model discovery. However, this approach fundamentally relies on an effective coordinate system in which the dynamics have a simple representation. In this work, we design a custom deep autoencoder network to discover a coordinate transformation into a reduced space where the dynamics may be sparsely represented. Thus, we simultaneously learn the governing equations and the associated coordinate system. We demonstrate this approach on several example high-dimensional systems with low-dimensional behavior. The resulting modeling framework combines the strengths of deep neural networks for flexible representation and sparse identification of nonlinear dynamics (SINDy) for parsimonious models. This method places the discovery of coordinates and models on an equal footing.

97 MATHEMATICS AND COMPUTING↗

A Quantum-Classical Collaborative Training Architecture Based on Quantum State Fidelity

Recent advancements have highlighted the limitations of current quantum systems, particularly the restricted number of qubits available on near-term quantum devices. This constraint greatly inhibits the range of applications that can leverage quantum computers. Moreover, as the available qubits increase, the computational complexity grows exponentially, posing additional challenges. Consequently, there is an urgent need to use qubits efficiently and mitigate both present limitations and future complexities. To address this, existing quantum applications attempt to integrate classical and quantum systems in a hybrid framework. In this study, we concentrate on quantum deep learning and introduce a collaborative classical-quantum architecture called co-TenQu. The classical component employs a tensor network for compression and feature extraction, enabling higher-dimensional data to be encoded onto logical quantum circuits with limited qubits. On the quantum side, we propose a quantum-state-fidelity-based evaluation function to iteratively train the network through a feedback loop between the two sides. co-TenQu has been implemented and evaluated with both simulators and the IBM-Q platform. Compared to state-of-the-art approaches, co-TenQu enhances a classical deep neural network by up to 41.72% in a fair setting. Additionally, it outperforms other quantum-based methods by up to 1.9 times and achieves similar accuracy while utilizing 70.59% fewer qubits.

42 ENGINEERING↗

Modeling Time-Dependent Surrogates of Additive-Manufactured Nuclear Fuels Processes

Additive manufacturing (AM) technology is being increasingly adopted in a wide variety of application areas for its ability to rapidly produce, prototype, and customize designs. Recently, a hybrid AM technique was successfully developed at Idaho National Laboratory (INL) to manufacture nuclear fuels [1]. Despite the advantages, this AM technique needs optimization due to defects from a highly complex melting and sintering process. The complex metallurgical phenomena during AM processes are strongly related to parameters such as applied laser power, traveling speed, and scan style, which could lead to differences in density, residual stress, crystallographic texture, and mechanical properties. In addition, stochastic variations in laser energy interaction and associated multiscale/multiphysics phenomena cause variations in microstructure evolution and mechanical properties. Currently, researchers at INL are focusing on developing a comprehensive modeling framework, leveraging INL’s simulation tools MOOSE/MARMOT/BISON/RAVEN [2-4] to describe all steps of this AM process across multiple length scales. Although this advanced framework plays a critical role in enabling enhancements to traditional trial and error approaches for design and optimization of nuclear fuel materials, it remains computationally intense, limiting its use in sensitivity and optimization analysis. In this case, an accurate and inexpensive surrogate becomes an effective tool for providing a tractable approximation of the underlying underline physics. Surrogate models generally not based on the physics of a system are purely mathematical models used to capture the relationships between specific system inputs and outputs. Popular approaches, including neural networks [5], response surfaces [6], and subspace-based reduced order models [7], have been applied to a wide range of disciplines, such as nuclear reactor design, aerospace design and automotive design. In this summary, we employ advanced time-dependent surrogate models such as high-dimensional model representation (HDMR) [8] and physics-informed deep neural network (PINNs) [9] to accelerate the design and optimization of AM process.

42 ENGINEERING↗

A deep learning approach to fast analysis of collective Thomson scattering spectra

Fast analysis of collective Thomson scattering ion acoustic wave features using a deep convolutional neural network model is presented. The network was trained from spectra to predict the plasma parameters, including ion velocities, population fractions, and ion and electron temperatures. A fully kinetic particle-in-cell simulation was used to model a laboratory astrophysics experiment and simulate a diagnostic image of the ion acoustic wave feature. Network predictions were compared with Bayesian inference of the plasma model parameters for both the simulated and experimentally measured images. Both approaches were fairly accurate predicting the simulated image and the network predictions matched a good portion of the Bayesian results for the experimentally measured image. The Bayesian approach is more robust to noise and motivates future work to train deep learning models with realistic noise. The advantage of the deep learning model is making thousands of predictions in a few hundred milliseconds, compared to a few seconds to minutes per prediction for the optimization and Bayesian approaches presented here. The results demonstrate promising capabilities of deep learning models to analyze Thomson data orders of magnitude faster than conventional methods when using the neural network for standalone analysis. If more rigorous analysis is needed, neural network predictions can be used to quickly initialize other optimization methods and increase chances of success. This is especially useful when the dataset becomes very large or highly dimensional and manually refining initial conditions for the entire dataset are no longer tractable.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Precision calibration of calorimeter signals in the ATLAS experiment using an uncertainty-aware neural network

The ATLAS experiment at the Large Hadron Collider explores the use of modern neural networks for a multi-dimensional calibration of its calorimeter signal defined by clusters of topologically connected cells (topo-clusters). The Bayesian neural network (BNN) approach not only yields a continuous and smooth calibration function that improves performance relative to the standard calibration but also provides uncertainties on the calibrated energies for each topo-cluster. The results obtained by using a trained BNN are compared to the standard local hadronic calibration and to a calibration provided by training a deep neural network. The uncertainties predicted by the BNN are interpreted in the context of a fractional contribution to the systematic uncertainties of the trained calibration. They are also compared to uncertainty predictions obtained from an alternative estimator employing repulsive ensembles.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Object Detection and Recognition with PointPillars in LiDAR Point Clouds – Comparisions

In the field of autonomous systems, neural networks have been leveraged for object detection and recognition in 2-dimensional images captured by cameras. Other types of sensors are available for sensing surroundings, including LiDAR sensors, and corresponding networks have been developed to perform detection and recognition in the point clouds generated by these sensors. The approaches are similar, both perform convolutions, but have distinct characteristics and challenges. In designing and configuring autonomous systems, a variety of LiDAR sensors are available, along with configurable deep neural networks to leverage their data. This work presents a review of the PointPillars network, an evolution of the seminal PointNet, comparing accuracy and training time relative to different LiDAR sensors, network and training parameters, CPU and GPU hardware, and the criticality of the use of reflective intensity as a feature. The value of using reflectivity as a predictive feature is explored and quantified to determine if it makes a significant difference in accuracy of the PointPillars network. Two separate LiDAR sensors are utilized, a 16-plane and a 32-plane, and corresponding accuracies and training times with the PointPillars network are evaluated.

LiDAR, machine learning, neural network, object re↗

Machine-Learning-Based High-Resolution Earthquake Catalog Reveals How Complex Fault Structures Were Activated during the 2016–2017 Central Italy Sequence

The 2016–2017 central Italy seismic sequence occurred on an 80 km long normal-fault system. The sequence initiated with the Mw 6.0 Amatrice event on 24 August 2016, followed by the Mw 5.9 Visso event on 26 October and the Mw 6.5 Norcia event on 30 October. We analyze continuous data from a dense network of 139 seismic stations to build a high-precision catalog of ~900,000 earthquakes spanning a 1 yr period, based on arrival times derived using a deep-neural-network-based picker. Our catalog contains an order of magnitude more events than the catalog routinely produced by the local earthquake monitoring agency. Aftershock activity reveals the geometry of complex fault structures activated during the earthquake sequence and provides additional insights into the potential factors controlling the development of the largest events. Activated fault structures in the northern and southern regions appear complementary to faults activated during the 1997 Colfiorito and 2009 L’Aquila sequences, suggesting that earthquake triggering primarily occurs on critically stressed faults. Delineated major fault zones are relatively thick compared to estimated earthquake location uncertainties, and a large number of kilometer-long faults and diffuse seismicity were activated during the sequence. These properties might be related to fault age, roughness, and the complexity of inherited structures. The rich details resolvable in this catalog will facilitate continued investigation of this energetic and well-recorded earthquake sequence.

58 GEOSCIENCES↗

Enabling large-scale viscoelastic calculations via neural network acceleration

One of the most significant challenges involved in efforts to understand the effects of repeated earthquake cycle activity is the computational costs of large-scale viscoelastic earthquake cycle models. Computationally intensive viscoelastic codes must be evaluated at thousands of times and locations, and as a result, studies tend to adopt a few fixed rheological structures and model geometries and examine the predicted time-dependent deformation over short (<10 years) time periods at a given depth after a large earthquake. Training a deep neural network to learn a computationally efficient representation of viscoelastic solutions, at any time, location, and for a large range of rheological structures, allows these calculations to be done quickly and reliably, with high spatial and temporal resolutions. We demonstrate that this machine learning approach accelerates viscoelastic calculations by more than 50,000%. Finally, this magnitude of acceleration will enable the modeling of geometrically complex faults over thousands of earthquake cycles across wider ranges of model parameters and at larger spatial and temporal scales than have been previously possible.

58 GEOSCIENCES↗

Deep inverse photonic design: A tutorial

Deep learning (DL) has been used to design deep neural networks (DNNs) which have recently been applied to solving inverse problems in artificial electromagnetic materials (AEMs). Although inverse problems are often ill-posed, and therefore are difficult to solve, deep inverse models (DIMs) have achieved impressive results often surpassing capabilities possible with other approaches. We overview the process of deep inverse learning applied to AEM problems, including the building of data sets, design of a forward model, and comparison of inverse approaches including limitations. We conclude by detailing some important outstanding issues of deep inverse design of AEMs, and present an outlook of this exciting field.

42 ENGINEERING↗

Evaluating Scalograms for Seismic Event Denoising

Denoising contaminated seismic signals for later processing is a fundamental problem in seismic signals analysis. The most straightforward denoising approach, using spectral filtering, is not effective when noise and seismic signal occupy the same frequency range. Neural network approaches have shown success denoising local signal when trained on short-time Fourier transform spectrograms (Zhu et al 2018; Tibi et al 2021). Scalograms, a wavelet-based transform, achieved ~15% better reconstruction as measured by dynamic time warping on a seismic waveform test set than spectrograms, suggesting their use as an alternative for denoising. We train a deep neural network on a scalogram dataset derived from waveforms recorded by the University of Utah Seismograph Stations network. We find that initial results are no better than a spectrogram approach, with additional overhead imposed by the significantly larger size of scalograms. A robust exploration of neural network hyperparameters and network architecture was not performed, which could be done in follow on work.

58 GEOSCIENCES↗

Toward The Development of Hailstorm Climatologies Derived From Reanalyses and Infared/Passive Microwave Satellite Imagers

Geostationary satellite imagers, such as those of the Geostationary Operational Environmental Satellite (GOES) and Meteosat series, provide both historical and near-real-time observations of cloud top patterns that are commonly associated with severe convection. Environmental conditions favorable for severe weather are thought to be represented well by reanalyses. Predicting exactly where convection and costly storm hazards like hail will occur using models or satellite imagery alone, however, is extremely challenging. The multivariate combination of satellite-observed cloud patterns with reanalysis environmental parameters, linked to United States Next Generation Weather Radar- (NEXRAD-) estimated Maximum Expected Size of Hail (MESH) using a deep neural network (DNN), enables estimation of potentially severe hail likelihood for any observed storm cell. These estimates are specifically designed to make hail likelihood distinctions based on satellite-indicated points of deep convection within environments favorable for storm development. We seek an approach that can be used to estimate climatological hailstorm frequency and risk throughout the historical satellite data record. This presentation demonstrates that statistical distributions of convective parameters from satellite and reanalysis show separation between non-severe/severe hailstorm classes for predictors including overshooting cloud top temperature and area characteristics, convective available potential energy, vertical wind shear, 500 hPa temperature, mid-level lapse rate, precipitable water, and convective inhibition. These complex, multivariate predictor relationships are exploited within a DNN to produce a hail likelihood metric with a critical success index of 0.504 and Heidke skill score of 0.403, which is exceptional among recent analogous hail studies. Furthermore, applications of the DNN to select case studies demonstrate good qualitative agreement between hail likelihood and MESH. These hail classifications are aggregated across an 11-year GOES-12/13 image database to derive a hail frequency and severity climatology, which denotes the Central Plains, the Midwest, and northwestern Mexico as being the most hail-prone regions within the domain studied. Opportunities for training and applying DNN-based hailstorm predictions to recently developed GOES-8/10/12/13/16 and Meteosat Second Generation convective storm detection and characterization climatologies over South America and South Africa, respectively, will also be presented.

Kristopher Bedka↗

Correspondence between neuroevolution and gradient descent

Abstract We show analytically that training a neural network by conditioned stochastic mutation or neuroevolution of its weights is equivalent, in the limit of small mutations, to gradient descent on the loss function in the presence of Gaussian white noise. Averaged over independent realizations of the learning process, neuroevolution is equivalent to gradient descent on the loss function. We use numerical simulation to show that this correspondence can be observed for finite mutations, for shallow and deep neural networks. Our results provide a connection between two families of neural-network training methods that are usually considered to be fundamentally different.

97 MATHEMATICS AND COMPUTING↗

Reconstructing the kinematics of deep inelastic scattering with deep learning

We introduce a method to reconstruct the kinematics of neutral-current deep inelastic scattering (DIS) using a deep neural network (DNN). Unlike traditional methods, it exploits the full kinematic information of both the scattered electron and the hadronic-final state, and it accounts for QED radiation by identifying events with radiated photons and event-level momentum imbalance. The method is studied with simulated events at HERA and the future Electron–Ion Collider (EIC). We show that the DNN method outperforms all the traditional methods over the full phase space, improving resolution and reducing bias. Our method has the potential to extend the kinematic reach of future experiments at the EIC, and thus their discovery potential in polarized and nuclear DIS.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

A machine learning approach to thermal conductivity modeling: A case study on irradiated uranium-molybdenum nuclear fuels

A deep neural network was developed for the purpose of predicting thermal conductivity with a case study performed on irradiated nuclear fuel. Traditional thermal conductivity modeling approaches rely on existing theoretical frameworks that describe known, relevant phenomena that govern the microstructural evolution processes during neutron irradiation (such as recrystallization, and pore size, distribution and morphology). Current empirical modeling approaches, however, do not represent all irradiation test data well. We develop a machine learning approach to thermal conductivity modeling that does not require a priori knowledge of a specific material microstructure and system of interest. Our approach allows re-searchers to probe dependency of thermal conductivity on a variety of reactor operating and material conditions. The purpose of building such a model is to allow for improved predictive capabilities linking structure-property-processing-performance relationships in the system of interest (here, irradiated nuclear fuel), which could lead to improved experimental test planning and characterization. The uranium-molybdenum system is the fuel system studied in this work, and historic irradiation test data is leveraged for model development. Our model achieved a mean absolute percent error of approximately 10%. Results indicate our model generalizes well to never before seen data, and thus use of deep learning methods for material property predictions from limited, historic irradiation data is a viable approach. This work challenges the current paradigm in materials science, where material property models are based on limited experimental data fitted to low-dimensionality phenomenological models, or by simulations in which fundamental equations are explicitly solved. The work presented here aims to demonstrate the potential and limitations of machine learning in the field of materials science and material property modeling.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Coarse-Grained Density Functional Theory Predictions via Deep Kernel Learning

Scalable electronic predictions are critical for soft materials design. Recently, the Electronic Coarse-Graining (ECG) method was introduced to renormalize all-atom quantum chemical (QC) predictions to coarse-grained (CG) resolutions using deep neural networks (DNNs). While DNNs can learn complex representations that prove challenging for kernel-based methods, they are susceptible to overfitting and the overconfidence of uncertainty estimations. Here, we develop ECG within a GPU-accelerated Deep Kernel Learning (DKL) framework to enable CG QC predictions using range-separated hybrid density functional theory (DFT), obtaining a 107 speedup relative to naive all-atom QC. By treating the predicted electronic properties as random Gaussian Processes, DKL incorporates CG mapping degeneracy by learning the distribution of electronic energies as a function of CG configuration. DKL-ECG accurately reproduces molecular orbital energies from range-separated DFT while facilitating efficient training via active learning using the uncertainties provided by DKL. Further, we show that while active learning algorithms enable efficient sampling of a more diverse configurational space relative to random sampling, all explored query methods exhibit comparable performance for the examined system. We attribute this result to the significant overlap of the feature space and output property distributions across multiple temperatures.

97 MATHEMATICS AND COMPUTING↗

Approximating Nash Equilibrium in Day-ahead Electricity Market Bidding with Multi-agent Deep Reinforcement Learning

In this paper, a day-ahead electricity market bidding problem with multiple strategic generation company (GEN-CO) bidders is studied. The problem is formulated as a Markov game model, where GENCO bidders interact with each other todevelop their optimal day-ahead bidding strategies. Considering unobservable information in the problem, a model-free and data-driven approach, known as multi-agent deep deterministic policy gradient (MADDPG), is applied for approximating the Nash equilibrium (NE) in the above Markov game. The MADDPG algorithm has the advantage of generalization due to the automatic feature extraction ability of the deep neural networks. The algorithm is tested on an IEEE 30-bus system with three competitive GENCO bidders in both an uncongested caseand a congested case. Comparisons with a truthful bidding strategy and state-of-the-art deep reinforcement learning methods including deep Q network and deep deterministic policy gradient (DDPG) demonstrate that the applied MADDPG algorithm can find a superior bidding strategy for all the market participants with increased profit gains. In addition, the comparison with a conventional model-based method shows that the MADDPG algorithm has higher computational efficiency, which is feasible for real-world applications.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Applications of data mining technology to enhance O and M: automatic planning of electrical isolation with deep learning - 255

Data mining technology is now readily available, thanks to high powered, low cost computing resource and useful coding tools. We study several applications of data mining to enhance operation and maintenance (O and M), such as abnormality prediction, efficiency diagnosis, document traceability, image recognition, etc. In field support, we try automatic planning of electrical isolation with deep learning. Currently, a skilled engineer plans electrical isolation procedure with hundreds of the circuit diagrams and the related documents, taking man-hours. If this task becomes automatic, it is very efficient. The automatic planning has two issues. One is much calculation time of electrical circuit simulator, searching billions of electrical conducting paths. The other is interpretation of human fuzzy information, such as ease of work. Deep learning is helpful to resolve them. We performed a model case study of the electrical isolation. We applied a deep neural network (DNN) for dropping in the calculation time. We trained the circuit diagrams and the responses of the circuit simulator to the DNN, constructing an efficient path search algorithm in the DNN. The calculation time of the DNN was shorter by a factor of 32 for the model case, compared with that of the electrical circuit simulator. We applied a deep Q-network (DQN) for the interpretation of fuzzy information. The DQN optimized isolation procedure from the viewpoint of the work load with the equipment layout, related doses, and values of wet bulb globe temperature (WBGT) to prevent heatstroke. (authors)

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗