Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Deep Neural Networks”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,477 records · Page 82

Slides for LDRD review presentation of 18A12-131FP, "Cyber-Secure Physical Layer Security in Millimeter Wave Communications"

Objective Enhance cyber security, protection, and assurance of advanced mmWave wireless communications in the presence of unauthorized devices and intruders with novel capabilities within the physical layer. Significance Wireless communications using millimeter waves (mmWave) in the GigaHertz (GHz) band is a key new technology that will play a major role in the 5th Generation (5G) wireless networks. Cyber-physical solutions for other frequency bands are not directly applicable to mmWave technologies. We propose to validate the hypotheses that: 1) communications can be better protected from eavesdropping and cyber intrusion with security features built within the mmWave physical layer, 2) User authentication can be conducted in a non-intrusive manner and with high probability of success by the use of physical channel variations of mmWaves which get scattered by a human body.

99 GENERAL AND MISCELLANEOUS↗

Algorithms to Improve Training for Deep Learning with Diabetic Retinopathy Images

This is a minisymposium presentation I plan to give at the SIAM conference on Computational Science and Engineering on March 1st, 2021. The work is based on research from my SDRD this year and describes a new pooling method we've developed called variable stride. The goal of the minisymposium is to highlight some of the ground-breaking work being done across the National Nuclear Security Administration by graduate students and Postdocs.

97 MATHEMATICS AND COMPUTING↗

Rotational Equivariance for Object Classification Using xView

With the recent addition of large, curated and labeled data sets to the remote sensing discipline, deep learning models have largely surpassed the performance of classical techniques. These deep models, typically Convolutional Neural Networks, are invariant to translation through the use of successive convolution layers which are themselves equivariant to translation. Further, the combination of multiple convolution and pooling layers means that in practice, the model is also approximately invariant to translation. However, until recently these models could only approach rotational invariance through data augmentation. Here we propose using a new model formulation which achieves rotational equaivariance without data augmentation for overhead imagery classification. We utilize the popular xView data set to compare the rotational equivariance formalization against a regular CNN and CNN with rotational data augmentation for the task of image classification.

Bynum, Lucius EJ↗

Full-field imaging learning machine (FILM)

A method of determining dynamic properties of a structure (linear or nonlinear) includes receiving spatio-temporal inputs, generating mode shapes and modal components corresponding to the spatio-temporal inputs using a trained deep complexity coding artificial neural network, and subsequently generating the dynamic properties by analyzing each modal component using a trained learning machine. A computing system for non-contact determination of dynamic properties of a structure includes a camera, a processor, and a memory including computer-executable instructions. When the instructions are executed, the system is caused to receive spatio-temporal image data, decompose the spatio-temporal image data into constituent manifold components using an autoencoder, and analyze the constituent manifold components using a trained learning machine to determine the dynamic properties.

Yang, Yongchao↗

A Study of Complex Deep Learning Networks on High Performance, Neuromorphic, and Quantum Computers

Current Deep Learning models use highly optimized convolutional neural networks (CNN) trained on large graphical processing units (GPU)-based computers with a fairly simple layered network topology, i.e., highly connected layers, without intra-layer connections. Complex topologies have been proposed, but are intractable to train on current systems. Building the topologies of the deep learning network requires hand tuning, and implementing the network in hardware is expensive in both cost and power.In this paper, we evaluate deep learning models using three different computing architectures to address these problems: quantum computing to train complex topologies, high performance computing (HPC) to automatically determine network topology, and neuromorphic computing for a low-power hardware implementation. Due to input size limitations of current quantum computers we use the MNIST dataset for our evaluation.The results show the possibility of using the three architectures in tandem to explore complex deep learning networks that are untrainable using a von Neumann architecture. We show that a quantum computer can find high quality values of intra-layer connections and weights, while yielding a tractable time result as the complexity of the network increases; a high performance computer can find optimal layer-based topologies; and a neuromorphic computer can represent the complex topology and weights derived from the other architectures in low power memristive hardware.This represents a new capability that is not feasible with current von Neumann architecture. It potentially enables the ability to solve very complicated problems unsolvable with current computing technologies.

97 MATHEMATICS AND COMPUTING↗

A Study of Complex Deep Learning Networks on High Performance, Neuromorphic, and Quantum Computers

Current Deep Learning models use highly optimized convolutional neural networks (CNN) trained on large graphical processing units (GPU)-based computers with a fairly simple layered network topology, i.e., highly connected layers, without intra-layer connections. Complex topologies have been proposed, but are intractable to train on current systems. Building the topologies of the deep learning network requires hand tuning, and implementing the network in hardware is expensive in both cost and power. In this paper, we evaluate deep learning models using three different computing architectures to address these problems: quantum computing to train complex topologies, high performance computing (HPC) to automatically determine network topology, and neuromorphic computing for a low-power hardware implementation. Due to input size limitations of current quantum computers we use the MNIST dataset for our evaluation. The results show the possibility of using the three architectures in tandem to explore complex deep learning networks that are untrainable using a von Neumann architecture. We show that a quantum computer can find high quality values of intra-layer connections and weights, while yielding a tractable time result as the complexity of the network increases; a high performance computer can find optimal layer-based topologies; and a neuromorphic computer can represent the complex topology and weights derived from the other architectures in low power memristive hardware. This represents a new capability that is not feasible with current von Neumann architecture. It potentially enables the ability to solve very complicated problems unsolvable with current computing technologies.

Potok, Thomas E↗

A multi-scale convolutional neural network for autonomous anomaly detection and classification in a laser powder bed fusion additive manufacturing process

In-situ detection of processing defects is a critical challenge for Laser Powder Bed Fusion Additive Manufacturing. Many of these defects are related to interactions between the recoater blade, which spreads the powder, and the powder bed. Here this work leverages Deep Learning, specifically a Convolutional Neural Network (CNN), for autonomous detection and classification of many of these spreading anomalies. Importantly, the input layer of the CNN is modified to enable the algorithm to learn both the appearance of the powder bed anomalies as well as key contextual information at multiple size scales. These modifications to the CNN architecture are shown to improve the flexibility and overall classification accuracy of the algorithm while mitigating many human biases. A case study is used to demonstrate the utility of the presented methodology and the overall performance is shown to be superior to that of methodologies previously reported by the authors.

36 MATERIALS SCIENCE↗

SA-GAT-SR: self-adaptable graph attention networks with symbolic regression for high-fidelity material property prediction

Recent advances in machine learning have demonstrated an enormous utility of deep learning approaches, particularly Graph Neural Networks (GNNs) for materials science. These methods have emerged as powerful tools for high-throughput prediction of material properties, offering a compelling enhancement and alternative to traditional first-principles calculations. While the community has predominantly focused on developing increasingly complex and universal models to enhance predictive accuracy, such approaches often lack physical interpretability and insights into materials behavior. Here, we introduce a novel computational paradigm—Self-Adaptable Graph Attention Networks integrated with Symbolic Regression (SA-GAT-SR)—that synergistically combines the predictive capability of GNNs with the interpretative power of symbolic regression. Our framework employs a self-adaptable encoding algorithm that automatically identifies and adjust attention weights so as to screen critical features from an expansive 180-dimensional feature space while maintaining O(n) computational scaling. The integrated SR module subsequently distills these features into compact analytical expressions that explicitly reveal quantum-mechanically meaningful relationships, achieving 23 × acceleration compared to conventional SR implementations that heavily rely on first-principle calculations-derived features as input. This work suggests a new framework in computational materials science, bridging the gap between predictive accuracy and physical interpretability, offering valuable physical insights into material behavior.

36 MATERIALS SCIENCE↗

Multi-Level Structural Damage Characterization Using Sparse Acoustic Sensor Networks and Knowledge Transferred Deep Learning

Standard structural health monitoring techniques face well-known difficulties for comprehensive defect diagnosis in real-world structures that have structural, material, or geometric complexity. This motivates the exploration of machine-learning-based structural health monitoring methods in complex structures. However, creating sufficient training data sets with various defects is an ongoing challenge for data-driven machine (deep) learning algorithms. The ability to transfer the knowledge of a trained neural network from one component to another or to other sections of the same component would drastically reduce the required training data set. Also, it would facilitate computationally inexpensive machine learning based inspection systems. In this work, a machine-learning-based multi-level damage characterization is demonstrated with the ability to transfer trained knowledge within the sparse sensor network. A novel network spatial assistance and an adaptive convolution technique are proposed for efficient knowledge transfer within the deep learning algorithm. Proposed structural health monitoring method is experimentally evaluated on an aluminum plate with artificially induced defects. It was observed that the method improves the performance of knowledge transferred damage characterization by 50% during localization and 24% during severity assessment. Further, experiments using time windows with and without multiple edge reflections are studied. Results reveal that multiply scattered waves contain rich and deterministic defect signatures that can be mined using deep learning neural networks, improving the accuracy of both identification and quantification. In the case of a fixed sensor network, using multiply scattered waves shows 100% prediction accuracy at all levels of damage characterization.

36 MATERIALS SCIENCE↗

Uncertainty quantification of a deep learning fuel property prediction model

Deep learning models are being widely used in the field of combustion. Given the black-box nature of typical neural network based models, uncertainty quantification (UQ) is critical to ensure the reliability of predictions as well as the training datasets, and for a principled quantification of noise and its various sources. Deep learning surrogate models for predicting properties of chemical compounds and mixtures have been recently shown to be promising for enabling data-driven fuel design and optimization, with the ultimate goal of improving efficiency and lowering emissions from combustion engines. In this study, UQ is performed for a multi-task deep learning model that simultaneously predicts the research octane number (RON), Motor Octane Number (MON), and Yield Sooting Index (YSI) of pure components and multicomponent blends. The deep learning model is comprised of three smaller networks: Extractor 1, Extractor 2, and Predictor, and a mixing operator. The molecular fingerprints of individual components are encoded via Extractor 1 and Extractor 2, the mixing operator generates fingerprints for mixtures/blends based on linear mixing operation, and the predictor maps the fingerprint to the target properties. Two different classes of UQ methods, Monte Carlo ensemble methods and Bayesian neural networks (BNNs), are employed for quantifying the epistemic uncertainty. Combinations of Bernoulli and Gaussian distributions with DropConnect and DropOut techniques are explored as ensemble methods. All the DropConnect, DropOut and Bayesian layers are applied to the predictor network. Aleatoric uncertainty is modeled by assuming that each data point has an independent uncertainty associated with it. The results of the UQ study are further analyzed to compare the performance of BNN and ensemble methods. Although this study is confined to UQ of fuel property prediction, the methodologies are applicable to other deep learning frameworks that are being widely used in the combustion community.

33 ADVANCED PROPULSION SYSTEMS↗

Causally‐Informed Deep Learning to Improve Climate Models and Projections

Abstract Climate models are essential to understand and project climate change, yet long‐standing biases and uncertainties in their projections remain. This is largely associated with the representation of subgrid‐scale processes, particularly clouds and convection. Deep learning can learn these subgrid‐scale processes from computationally expensive storm‐resolving models while retaining many features at a fraction of computational cost. Yet, climate simulations with embedded neural network parameterizations are still challenging and highly depend on the deep learning solution. This is likely associated with spurious non‐physical correlations learned by the neural networks due to the complexity of the physical dynamical system. Here, we show that the combination of causality with deep learning helps removing spurious correlations and optimizing the neural network algorithm. To resolve this, we apply a causal discovery method to unveil causal drivers in the set of input predictors of atmospheric subgrid‐scale processes of a superparameterized climate model in which deep convection is explicitly resolved. The resulting causally‐informed neural networks are coupled to the climate model, hence, replacing the superparameterization and radiation scheme. We show that the climate simulations with causally‐informed neural network parameterizations retain many convection‐related properties and accurately generate the climate of the original high‐resolution climate model, while retaining similar generalization capabilities to unseen climates compared to the non‐causal approach. The combination of causal discovery and deep learning is a new and promising approach that leads to stable and more trustworthy climate simulations and paves the way toward more physically‐based causal deep learning approaches also in other scientific disciplines.

Meteorology & Atmospheric Sciences↗

Time-lapse seismic inversion for CO 2 saturation with SeisCO2Net: An application to Frio-II site

Seismic monitoring of geological CO 2 storage (GCS) involves highly nonlinear seismic inversion and petrophysical inversion, making it challenging to estimate CO 2 volume efficiently and detect possible early CO 2 leakages. Deep learning (DL) using convolutional neural networks (CNNs) has shown promise in solving highly nonlinear seismic inversion problems. However, direct estimation of CO 2 plume extent/saturation from time-lapse seismic gathers using DL is still underexplored, with no reported field applications to date. The investigation of field data is primarily hindered by scarcity of field data for neural network training. Other obstacles include highly nonlinear seismic-petrophysics inverse relationship, and presence of noise in field seismic data. We introduce SeisCO2Net, a deep CNN that predicts CO 2 saturation maps directly from time-lapse full waveform shot gathers. For training, we use site-specific geological information, fluid flow physics, rock physics, and seismic modeling to generate synthetic datasets that closely resemble the CO 2 storage site. Synthetic tests show promising results, inspiring us to apply SeisCO2Net's trained weights on field data collected at Frio-II GCS site by leveraging transfer learning principles. As reference, we compare SeisCO2Net's predicted CO 2 saturation maps with results obtained from physics-based inversion. Our analyses show both methods display similar CO 2 plume shapes, reasonable CO 2 plume characteristics, and comparable saturation values. Our results suggest pre-training CNNs on physics-informed synthetic datasets and then applying the learned weights to field data is a viable approach to estimating field CO 2 saturation. This method effectively addresses the scarcity of field training data, thus encouraging the feasibility of long-term GCS monitoring.

58 GEOSCIENCES↗

Understanding the Influence of Receptive Field and Network Complexity in Neural Network-Guided TEM Image Analysis

Abstract Trained neural networks are promising tools to analyze the ever-increasing amount of scientific image data, but it is unclear how to best customize these networks for the unique features in transmission electron micrographs. Here, we systematically examine how neural network architecture choices affect how neural networks segment, or pixel-wise separate, crystalline nanoparticles from amorphous background in transmission electron microscopy (TEM) images. We focus on decoupling the influence of receptive field, or the area of the input image that contributes to the output decision, from network complexity, which dictates the number of trainable parameters. For low-resolution TEM images which rely on amplitude contrast to distinguish nanoparticles from background, we find that the receptive field does not significantly influence segmentation performance. On the other hand, for high-resolution TEM images which rely on both amplitude and phase-contrast changes to identify nanoparticles, receptive field is an important parameter for increased performance, especially in images with minimal amplitude contrast. Rather than depending on atom or nanoparticle size, the ideal receptive field seems to be inversely correlated to the degree of nanoparticle contrast in the image. Our results provide insight and guidance as to how to adapt neural networks for applications with TEM datasets.

42 ENGINEERING↗

Detection of Hail Storms in Radar Imagery Using Deep Learning

In 2016, hail was responsible for 3.5 billion and 23 million dollars in damage to property and crops, respectively, making it the second costliest weather phenomenon in the United States. In an effort to improve hail-prediction techniques and reduce the societal impacts associated with hail storms, we propose a deep learning technique that leverages radar imagery for automatic detection of hail storms. The technique is applied to radar imagery from 2011 to 2016 for the contiguous United States and achieved a precision of 0.848. Hail storms are primarily detected through the visual interpretation of radar imagery (Mrozet al., 2017). With radars providing data every two minutes, the detection of hail storms has become a big data task. As a result, scientists have turned to neural networks that employ computer vision to identify hail-bearing storms (Marzbanet al., 2001). In this study, we propose a deep Convolutional Neural Network (ConvNet) to understand the spatial features and patterns of radar echoes for detecting hailstorms.

natural hazard↗

GrAMME: Semisupervised Learning Using Multilayered Graph Attention Models

Modern data analysis pipelines are becoming increasingly complex due to the presence of multiview information sources. While graphs are effective in modeling complex relationships, in many scenarios, a single graph is rarely sufficient to succinctly represent all interactions, and hence, multilayered graphs have become popular. Though this leads to richer representations, extending solutions from the single-graph case is not straightforward. Consequently, there is a strong need for novel solutions to solve classical problems, such as node classification, in the multilayered case. In this article, we consider the problem of semisupervised learning with multilayered graphs. Though deep network embeddings, e.g., DeepWalk, are widely adopted for community discovery, we argue that feature learning with random node attributes, using graph neural networks, can be more effective. To this end, we propose to use attention models for effective feature learning and develop two novel architectures, GrAMME-SG and GrAMME-Fusion, that exploit the interlayer dependences for building multilayered graph embeddings. Using empirical studies on several benchmark data sets, we evaluate the proposed approaches and demonstrate significant performance improvements in comparison with the state-of-the-art network embedding strategies. The results also show that using simple random features is an effective choice, even in cases where explicit node attributes are not available.

97 MATHEMATICS AND COMPUTING↗

Identifying Sample Provenance From SEM/EDS Automated Particle Analysis via Few-Shot Learning Coupled With Similarity Graph Clustering

Automated particle analysis (APA) provides a vast amount of compositional data via energy-dispersive X-ray spectroscopy along with size and shape data via scanning electron microscopy for individual particles in a sample. In many instances, APA data are leveraged to support identification of the source of a sample based on the detection of particles of a specific composition. Often, the particles that provide context make up a minuscule portion of the sample. Additionally, the interpretation of complex samples can be difficult due to the diversity of compositions both in the mixture and within a particle. In this work, we demonstrate a method to compute and cluster similarity graphs that describe inter-particle relationships within a sample using a multi-modal few-shot learning neural network. Here, as a proof-of-concept, we show that samples known to have been exposed to gunshot residue can be distinguished from samples occasionally mistaken for gunshot residue. Our workflow builds upon standard APA techniques and data processing methods to unveil additional information in a readily interpretable and quantitatively comparable format.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗