Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Deep Neural Networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,045 records · Page 58

Classify QCD phase transition with deep learning

The state-of-the-art pattern recognition method in machine learning (deep convolution neural network) is used to identify the equation of state (EoS) employed in the relativistic hydrodynamic simulations of heavy ion collisions. High-level correlations of particle spectra in transverse momentum and azimuthal angle learned by the network act as an effective EoS-meter in deciphering the nature of the phase transition in QCD. The EoS-meter is model independent and insensitive to other simulation inputs including the initial conditions and shear viscosity for hydrodynamic simulations. Through this study we demonstrate that there is a traceable encoder of the dynamical information from the phase structure that survives the evolution and exists in the final snapshot of heavy ion collisions and one can exclusively and effectively decode these information from the highly complex final output with machine learning when traditional methods fail. Besides the deep neural network, the performance of traditional machine learning classifiers are also provided.

74 ATOMIC AND MOLECULAR PHYSICS↗

DETECTING FIRE WITH MACHINE LEARNING-ENABLED VISUAL MONITORING FOR NUCLEAR POWER PLANT ENVIRONMENTS

Nuclear power plants are experiencing significant cost challenges to remain competitive with other energy-generation utilities. Unlike other industries, the cost of operation and maintenance activities is mostly attributed to workforce costs. To mitigate this, nuclear power plant stakeholders are increasingly interested in the development and deployment of machine learning methods to potentially automate or augment manually intensive tasks to reduce costs, especially for monitoring activities. One monitoring function that is visually demanding and that can occur frequently to meet the requirements of a fire protection program is visually monitoring an area for fire occurrence. Currently, fire watch activities consist of a worker physically stationed at a given location with the sole responsibility of observing a given area to ensure a fire is detected and mitigated promptly. This effort focused on the development and evaluation of a suitable deep convolutional neural network to classify individual video frames at a sub-second frequency for the occurrence of “fire” and “no fire” in varying industrial environments similar to nuclear power plants. It is believed that a trained neural network model could be integrated with existing facility video surveillance camera feeds to generate alerts when fire inferences occur in individual frames captured at sub-second temporal resolutions. Extensive effort was dedicated to identifying and curating suitable imagery training data representing varying environments and scene settings with and without flame features to maximize generalization in nuclear power plant environments. The data collection effort resulted in the aggregation of a large, labeled image library exceeding 12,000 images to support model training for diverse industrial environments. A deep neural network model incorporating parallel multi-scale capabilities was developed and trained to support accurate image-based detection of flame incidents of varying sizes and spectral feature properties within heterogeneous scenes. Analysis results show that the trained model can achieve high inference accuracy despite heterogeneous scene environments and components. Testing accuracy exceeded 95.0 percent with very low false positive and false negative inferences.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Neural network based satellite tracking for deep space applications

The objective of this paper is to provide a survey of neural network trends as applied to the tracking of spacecrafts in deep space at Ka-band under various weather conditions and examine the trade-off between tracing accuracy and communication link performance.

tracking deep space monopulse neural network weath↗

NeMO-Net – The Neural Multi-Modal Observation & Training Network for Global Coral Reef Assessment

We present NeMO-Net, the Srst open-source deep convolutional neural network (CNN) and interactive learning and training software aimed at assessing the present and past dynamics of coral reef ecosystems through habitat mapping into 10 biological and physical classes. Shallow marine systems, particularly coral reefs, are under significant pressures due to climate change, ocean acidification, and other anthropogenic pressures, leading to rapid, often devastating changes, in these fragile and diverse ecosystems. Historically, remote sensing of shallow marine habitats has been limited to meter-scale imagery due to the optical effects of ocean wave distortion, refraction, and optical attenuation. NeMO-Net combines 3D cm-scale distortion-free imagery captured using NASA FluidCam and Fluid lensing remote sensing technology with low resolution airborne and spaceborne datasets of varying spatial resolutions, spectral spaces, calibrations, and temporal cadence in a supercomputer-based machine learning framework. NeMO-Net augments and improves the benthic habitat classification accuracy of low-resolution datasets across large geographic ad temporal scales using high-resolution training data from FluidCam.NeMO-Net uses fully convolutional networks based upon ResNet and ReSneNet to perform semantic segmentation of remote sensing imagery of shallow marine systems captured by drones, aircraft, and satellites, including WorldView and Sentinel. Deep Laplacian Pyramid Super-Resolution Networks (LapSRN) alongside Domain Adversarial Neural Networks (DANNs) are used to reconstruct high resolution information from low resolution imagery, and to recognize domain-invariant features across datasets from multiple platforms to achieve high classification accuracies, overcoming inter-sensor spatial, spectral and temporal variations.Finally, we share our online active learning and citizen science platform, which allows users to provide interactive training data for NeMO-Net in 2D and 3D, integrated within a deep learning framework. We present results from the PaciSc Islands including Fiji, Guam and Peros Banhos 1 1 2 1 3 1 where 24-class classification accuracy exceeds 91%.

Chirayath, Ved↗

Reducing model bias in a deep learning classifier using domain adversarial neural networks in the MINERvA experiment

We present a simulation-based study using deep convolutional neural networks (DCNNs) to identify neutrino interaction vertices in the MINERvA passive targets region, and illustrate the application of domain adversarial neural networks (DANNs) in this context. DANNs are designed to be trained in one domain (simulated data) but tested in a second domain (physics data) and utilize unlabeled data from the second domain so that during training only features which are unable to discriminate between the domains are promoted. MINERvA is a neutrino-nucleus scattering experiment using the NuMI beamline at Fermilab. A-dependent cross sections are an important part of the physics program, and these measurements require vertex finding in complicated events. To illustrate the impact of the DANN we used a modified set of simulation in place of physics data during the training of the DANN and then used the label of the modified simulation during the evaluation of the DANN. In conclusion, we find that deep learning based methods offer significant advantages over our prior track-based reconstruction for the task of vertex finding, and that DANNs are able to improve the performance of deep networks by leveraging available unlabeled data and by mitigating network performance degradation rooted in biases in the physics models used for training.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Challenges and Opportunities in Deep Reinforcement Learning With Graph Neural Networks: A Comprehensive Review of Algorithms and Applications

Deep reinforcement learning (DRL) has empowered a variety of artificial intelligence fields, including pattern recognition, robotics, recommendation-systems, and gaming. Similarly, graph neural networks (GNN) have also demonstrated their superior performance in supervised learning for graph-structured data. In recent times, the fusion of GNN with DRL for graph-structured environments has attracted a lot of attention. Here, this paper provides a comprehensive review of these hybrid works. These works can be classified into two categories: (1) algorithmic enhancement, where DRL and GNN complement each other for better utility; (2) application-specific enhancement, where DRL and GNN support each other. This fusion effectively addresses various complex problems in engineering and life sciences. Based on the review, we further analyze the applicability and benefits of fusing these two domains, especially in terms of increasing generalizability and reducing computational complexity. Finally, the key challenges in integrating DRL and GNN, and potential future research directions are highlighted, which will be of interest to the broader machine learning community.

97 MATHEMATICS AND COMPUTING↗

Identification of Abnormal System Noise Temperature Patterns in Deep Space Network Antennas Using Neural Network Trained Fuzzy Logic

This paper presents the development of a fuzzy logic function trained by an artificial neural network to classify the system noise temperature (SNT) of antennas in the NASA Deep Space Network (DSN). The SNT data were classified into normal, marginal, and abnormal classes. The irregular SNT pattern was further correlated with link margin and weather data. A reasonably good correlation is detected among high SNT, low link margin and the effect of bad weather; however we also saw some unexpected non-correlations which merit further study in the future.

pattern identification↗

Variational Neural-Network Ansatz for Continuum Quantum Field Theory

Physicists dating back to Feynman have lamented the difficulties of applying the variational principle to quantum field theories. In nonrelativistic quantum field theories, the challenge is to parametrize and optimize over the infinitely many n-particle wave functions comprising the state’s Fock-space representation. Here we approach this problem by introducing neural-network quantum field states, a deep learning ansatz that enables application of the variational principle to nonrelativistic quantum field theories in the continuum. Our ansatz uses the Deep Sets neural network architecture to simultaneously parametrize all of the n-particle wave functions comprising a quantum field state. We employ our ansatz to approximate ground states of various field theories, including an inhomogeneous system and a system with long-range interactions, thus demonstrating a powerful new tool for probing quantum field theories.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Learning and discovering multiple solutions using physics-informed neural networks with random initialization and deep ensemble

In this work we explore the capability of physics-informed neural networks (PINNs) to discover multiple solutions. Many real-world phenomena governed by nonlinear differential equations (DEs), such as fluid flow, exhibit multiple solutions under the same conditions, yet capturing this solution multiplicity remains a significant challenge. A key difficulty lies in providing appropriate initial conditions or guesses, as widely used time-marching schemes and Newton’s method are highly sensitive to these choices when solving complex computational problems. While machine learning models, particularly PINNs, have shown promise in solving DEs, their ability to capture multiple solutions remains underexplored. In this work, we propose a simple and practical approach using PINNs to learn and discover multiple solutions. We first demonstrate that PINNs, when combined with random initialization and deep ensemble method—originally developed for uncertainty quantification—can effectively uncover multiple solutions to nonlinear ordinary and partial DEs. Although training large ensembles of PINNs may appear computationally demanding, this can be done efficiently using vectorization techniques supported by modern deep learning frameworks, allowing many networks to be trained simultaneously. Our approach highlights the critical role of initialization in shaping solution diversity, addressing an often-overlooked aspect of machine learning for scientific computing. Furthermore, we propose utilizing PINN-generated solutions as initial conditions or initial guesses for conventional numerical solvers to enhance accuracy and efficiency in capturing multiple solutions. Extensive numerical experiments, including the Allen–Cahn equation and cavity flow, where our approach successfully identifies both stable and unstable solutions, validate the effectiveness of our method. These findings establish a general and efficient framework for addressing solution multiplicity in nonlinear DEs.

97 MATHEMATICS AND COMPUTING↗

Embedded training of neural-network subgrid-scale turbulence models

We report that the weights of a deep neural-network model are optimized in conjunction with the governing flow equations to provide a model for subgrid-scale stresses in a temporally developing plane turbulent jet at Reynolds number Re 0 = 6000 . The objective function for training is first based on the instantaneous filtered velocity fields from a corresponding direct numerical simulation, and the training is by a stochastic gradient descent method, which uses the adjoint Navier-Stokes equations to provide the end-to-end sensitivities of the model weights to the velocity fields. In-sample and out-of-sample testing on multiple dual-jet configurations show that its required mesh density in each coordinate direction for prediction of mean flow, Reynolds stresses, and spectra is half that needed by the dynamic Smagorinsky model for comparable accuracy. The same neural-network model trained directly to match filtered subgrid-scale stresses, without the constraint of being embedded within the flow equations during the training, fails to provide a qualitatively correct prediction. The coupled formulation is generalized to train based only on mean-flow and Reynolds stresses, which are more readily available in experiments. The mean-flow training provides a robust model, which is important, though a somewhat less accurate prediction for the same coarse meshes, as might be anticipated due to the reduced information available for training in this case. The anticipated advantage of the formulation is that the inclusion of resolved physics in the training increases its capacity to extrapolate. This is assessed for the case of passive scalar transport, for which it outperforms established models due to improved mixing predictions.

42 ENGINEERING↗

Exploring Physics of Ferroelectric Domain Walls in Real Time: Deep Learning Enabled Scanning Probe Microscopy

The functionality of ferroelastic domain walls in ferroelectric materials is explored in real-time via the in situ implementation of computer vision algorithms in scanning probe microscopy (SPM) experiment. The robust deep convolutional neural network (DCNN) is implemented based on a deep residual learning framework (Res) and holistically nested edge detection (Hed), and ensembled to minimize the out-of-distribution drift effects. The DCNN is implemented for real-time operations on SPM, converting the data stream into the semantically segmented image of domain walls and the corresponding uncertainty. Further the pre-defined experimental workflows perform piezoresponse spectroscopy measurement on thus discovered domain walls, and alternating high- and low-polarization dynamic (out-of-plane) ferroelastic domain walls in a PbTiO 3 (PTO) thin film and high polarization dynamic (out-of-plane) at short ferroelastic walls (compared with long ferroelastic walls) in a lead zirconate titanate (PZT) thin film is reported. This work establishes the framework for real-time DCNN analysis of data streams in scanning probe and other microscopies and highlights the role of out-of-distribution effects and strategies to ameliorate them in real time analytics.

36 MATERIALS SCIENCE↗

DeepONet-grid-UQ: A trustworthy deep operator framework for predicting the power grid’s post-fault trajectories

This paper proposes a novel data-driven method for the reliable prediction of the power grid’s post-fault trajectories, i.e., the power grid’s dynamic response after a disturbance or fault. Here, the proposed method is based on the recently proposed concept of Deep Operator Networks (DeepONets). Unlike traditional neural networks that learn to approximate functions, DeepONets are designed to approximate nonlinear operators, i.e., mappings between infinite-dimensional spaces. Under this operator framework, we design a novel and efficient DeepONet that (i) takes as inputs the trajectories collected before and during the fault and (ii) outputs the predicted post-fault trajectories. In addition, we endow our method with the much-needed ability to balance efficiency with reliable/trustworthy predictions via uncertainty quantification. To this end, we propose and compare two novel methods that enable quantifying the predictive uncertainty. First, we propose a Bayesian DeepONet (B-DeepONet) that uses stochastic gradient Hamiltonian Monte-Carlo to sample from the posterior distribution of the DeepONet trainable parameters. Then, we design a Probabilistic DeepONet (Prob-DeepONet) that uses a probabilistic training strategy to enable quantifying uncertainty at virtually no extra computational cost. Finally, we validate the proposed methods’ predictive power and uncertainty quantification capability using the New York-New England power grid model.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Determination of Infinite Dilution Activity Coefficients of Molecular Solutes in Ionic Liquids and Deep Eutectic Solvents by Factorization-Machine-Based Neural Networks

Widely known as “green solvents,” ionic liquids (ILs) and deep eutectic solvents (DESs) have been used as substitutes for traditional organic solvents in separation science. To achieve better separations using ILs and DESs, this work aimed to predict the infinite dilution activity coefficients (IDACs) of molecular solutes in these solvents with the state-of-the-art factorization-machine-based neural network (DeepFM). DeepFM combines the benefits of factorization machines and deep neural networks to learn both low-order and high-order interactions among features. The IDAC prediction model was established with 52,372 experimental IDAC datapoints including 260 solvents (252 ILs and 8 DESs) and 112 molecular solutes collected at various temperatures from 288.15 to 428.15 K. Chemical information describing the ILs, DESs, and molecular solutes was included in the IDAC prediction model, including chemical functional groups, molecular weights, and Abraham solvation parameters. The IDAC prediction model showed an improved accuracy compared with alternative models; additionally, DESs were included in the IDAC prediction model for the first time. Here, the model will reduce the energy and resources needed to optimize the selection of ILs and DESs for specific separations, which will promote the development of these green solvents for sustainable chemical processes.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Inverse Analysis with Variational Autoencoders: A Comparison of Shallow and Deep Networks

Inverse problems are applied to determine unknown properties by matching observational data with a physical model that often takes many parameters as input. To overcome the underconstrained nature of inverse problems and achieve good performance, an approach is presented involving regularization with a technique known as a variational autoencoder (VAE), which is trained to map a high-dimensional parameter space with a complex structure to a low-dimensional latent space with a simple structure. We apply this approach to unconditioned realizations of the parameters (heterogeneous hydraulic fields) for a hydrogeological inverse problem. Two types of hydraulic conductivity fields are used to evaluate the characterization for the different levels of heterogeneity complexity of the physical inputs. This approach keeps the computational cost of generating the training data low. The reason is unconditioned realizations neither rely on the observational data used to perform the inverse analysis nor require any groundwater flow model forward runs. In addition, this approach applies regularization on a low-dimensional latent space from the VAE and increases optimization efficiency through automatic differentiation. Furthermore, two different neural network (NN) structures are tested for their utility in using the VAE for inverse analysis. The performance of a deep, convolutional neural network strongly depends on the dimensionality of the latent space, which requires tuning. In contrast, a shallow, dense neural network provides consistently accurate characterization without tuning. Furthermore, our approach evaluates the advantages of the shallow, dense neural network over the deep, convolutional one and enables future application to a wide range of inverse problems.

97 MATHEMATICS AND COMPUTING↗

Provable Repair of Vision Transformers

Vision Transformers have emerged as state-of-the-art image recognition tools, but may still exhibit incorrect behavior. Incorrect image recognition can have disastrous consequences in safety-critical real-world applications such as self-driving automobiles. In this paper, we present Provable Repair of Vision Transformers (PRoViT), a provable repair approach that guarantees the correct classification of images in a repair set for a given Vision Transformer without modifying its architecture. PRoViT avoids negatively affecting correctly classified images (drawdown) by minimizing the changes made to the Vision Transformer’s parameters and original output. Here, we observe that for Vision Transformers, unlike for other architectures such as ResNet or VGG, editing just the parameters in the last layer achieves correctness guarantees and very low drawdown. We introduce a novel method for editing these last-layer parameters that enables PRoViT to efficiently repair state-of-the-art Vision Transformers for thousands of images, far exceeding the capabilities of prior provable repair approaches.

97 MATHEMATICS AND COMPUTING↗

Ultrafast Bragg coherent diffraction imaging of epitaxial thin films using deep complex-valued neural networks

Abstract Domain wall structures form spontaneously due to epitaxial misfit during thin film growth. Imaging the dynamics of domains and domain walls at ultrafast timescales can provide fundamental clues to features that impact electrical transport in electronic devices. Recently, deep learning based methods showed promising phase retrieval (PR) performance, allowing intensity-only measurements to be transformed into snapshot real space images. While the Fourier imaging model involves complex-valued quantities, most existing deep learning based methods solve the PR problem with real-valued based models, where the connection between amplitude and phase is ignored. To this end, we involve complex numbers operation in the neural network to preserve the amplitude and phase connection. Therefore, we employ the complex-valued neural network for solving the PR problem and evaluate it on Bragg coherent diffraction data streams collected from an epitaxial La 2-x Sr x CuO 4 (LSCO) thin film using an X-ray Free Electron Laser (XFEL). Our proposed complex-valued neural network based approach outperforms the traditional real-valued neural network methods in both supervised and unsupervised learning manner. Phase domains are also observed from the LSCO thin film at an ultrafast timescale using the complex-valued neural network.

77 NANOSCIENCE AND NANOTECHNOLOGY↗

Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams

Situational awareness data is a prototypical example of structured heterogeneous streaming data with asynchronous timing that can quickly scale beyond the cognitive power of a human analyst. We present deep learning methods that can train in real time from unlabeled streams and provide probabilistic anomaly scores. We compare our novel deep and recurrent neural network models to three baseline systems using threat detection recall on the CERT Insider Threat Dataset v6.2. In our experiments, our deep neural network and our recurrent neural network systems outperformed all baselines, and we found that using aggregated activity count features yielded the best performance. Our best model produces on average an anomaly score of 95.53 percentile for the day, suggesting it can facilitate a great reduction in analyst workload. Our models are also interpretable, in that they break down the anomaly score into individual features of user behavior, which could further aid analysts in reviewing potential cases of insider threat.

Tuor, Aaron R.↗