Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “deep”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,621 records · Page 90

Deep Potential Molecular Dynamics: A Scalable Model with the Accuracy of Quantum Mechanics

We introduce a scheme for molecular simulations, the deep potential molecular dynamics (DPMD) method, based on a many-body potential and interatomic forces generated by a carefully crafted deep neural network trained with ab initio data. The neural network model preserves all the natural symmetries in the problem. It is first-principles based in the sense that there are no ad hoc components aside from the network model. We show that the proposed scheme provides an efficient and accurate protocol in a variety of systems, including bulk materials and molecules. In all these cases, DPMD gives results that are essentially indistinguishable from the original data, at a cost that scales linearly with system size.

74 ATOMIC AND MOLECULAR PHYSICS↗

Maximum Entropy Principle in Deep Thermalization and in Hilbert-Space Ergodicity

We report universal statistical properties displayed by ensembles of pure states that naturally emerge in quantum many-body systems. Specifically, two classes of state ensembles are considered: those formed by (i) the temporal trajectory of a quantum state under unitary evolution or (ii) the quantum states of small subsystems obtained by partial, local projective measurements performed on their complements. These cases, respectively, exemplify the phenomena of “Hilbert-space ergodicity” and “deep thermalization.” In both cases, the resultant ensembles are defined by a simple principle: The distributions of pure states have maximum entropy, subject to constraints such as energy conservation, and effective constraints imposed by thermalization. We present and numerically verify quantifiable signatures of this principle by deriving explicit formulas for all statistical moments of the ensembles, proving the necessary and sufficient conditions for such universality under widely accepted assumptions, and describing their measurable consequences in experiments. We further discuss information-theoretic implications of the universality: Our ensembles have maximal information content while being maximally difficult to interrogate, establishing that generic quantum state ensembles that occur in nature hide (scramble) information as strongly as possible. Our results generalize the notions of Hilbert-space ergodicity to time-independent Hamiltonian dynamics and deep thermalization from infinite to finite effective temperature. Our work presents new perspectives to characterize and understand universal behaviors of quantum dynamics using statistical and information-theoretic tools.

Eigenstate thermalization↗

Matrix-Model Simulations Using Quantum Computing, Deep Learning, and Lattice Monte Carlo

Matrix quantum mechanics plays various important roles in theoretical physics, such as a holographic description of quantum black holes, and it underpins the only practical numerical approach to the study of complex high-dimensional supergravity theories. Understanding quantum black holes and the role of entanglement in a holographic setup is of paramount importance for the realization of a quantum theory of gravity. Moreover, a complete numerical understanding of the holographic duality and the emergence of geometric space-time features from microscopic degrees of freedom could pave the way for new discoveries in quantum information science. Euclidean lattice Monte Carlo simulations are the de facto numerical tool for understanding the spectrum of large matrix models and have been used to test the holographic duality. However, they are not tailored to extract dynamical properties or even the quantum wave function of the ground state of matrix models. Quantum computing and deep learning provide potentially useful approaches to study the dynamics of matrix quantum mechanics. If successful in the context of matrix models, these rapidly improving numerical techniques could become the new Swiss army knife of quantum gravity practitioners. In this paper, we perform the first systematic survey for quantum computing and deep-learning approaches to matrix quantum mechanics, comparing them to lattice Monte Carlo simulations. These provide baseline benchmarks before addressing more complicated problems. In particular, we test the performance of each method by calculating the low-energy spectrum.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Enhancing synchrotron radiation micro-CT images using deep learning: an application of Noise2Inverse on bone imaging

In bone-imaging research, in situ synchrotron radiation micro-computed tomography (SRµCT) mechanical tests are used to investigate the mechanical properties of bone in relation to its microstructure. Low-dose computed tomography (CT) is used to preserve bone's mechanical properties from radiation damage, though it increases noise. To reduce this noise, the self-supervised deep learning method Noise2Inverse was used on low-dose SRµCT images where segmentation using traditional thresholding techniques was not possible. Simulated-dose datasets were created by sampling projection data at full, one-half, one-third, one-fourth and one-sixth frequencies of an in situ SRµCT mechanical test. After convolutional neural networks were trained, Noise2Inverse performance on all dose simulations was assessed visually and by analyzing bone microstructural features. Visually, high image quality was recovered for each simulated dose. Lacunae volume, lacunae aspect ratio and mineralization distributions shifted slightly in full, one-half and one-third dose network results, but were distorted in one-fourth and one-sixth dose network results. Following this, new models were trained using a larger dataset to determine differences between full dose and one-third dose simulations. Significant changes were found for all parameters of bone microstructure, indicating that a separate validation scan may be necessary to apply this technique for microstructure quantification. Noise present during data acquisition from the testing setup was determined to be the primary source of concern for Noise2Inverse viability. While these limitations exist, incorporating dose calculations and optimal imaging parameters enables self-supervised deep learning methods such as Noise2Inverse to be integrated into existing experiments to decrease radiation dose.

Obata, Yoshihiro (ORCID:0000000303659129)↗

BraggNN : fast X-ray Bragg peak analysis using deep learning

X-ray diffraction based microscopy techniques such as high-energy diffraction microscopy (HEDM) rely on knowledge of the position of diffraction peaks with high precision. These positions are typically computed by fitting the observed intensities in detector data to a theoretical peak shape such as pseudo-Voigt. As experiments become more complex and detector technologies evolve, the computational cost of such peak-shape fitting becomes the biggest hurdle to the rapid analysis required for real-time feedback in experiments. To this end, we propose BraggNN, a deep-learning based method that can determine peak positions much more rapidly than conventional pseudo-Voigt peak fitting. When applied to a test dataset, peak center-of-mass positions obtained from BraggNN deviate less than 0.29 and 0.57 pixels for 75 and 95% of the peaks, respectively, from positions obtained using conventional pseudo-Voigt fitting (Euclidean distance). When applied to a real experimental dataset and using grain positions from near-field HEDM reconstruction as ground-truth, grain positions using BraggNN result in 15% smaller errors compared with those calculated using pseudo-Voigt. Recent advances in deep-learning method implementations and special-purpose model inference accelerators allow BraggNN to deliver enormous performance improvements relative to the conventional method, running, for example, more than 200 times faster on a consumer-class GPU card with out-of-the-box software.

36 MATERIALS SCIENCE↗

Crystal structure and RNA-binding properties of an Hfq homolog from the deep-branching Aquificae: conservation of the lateral RNA-binding mode

The host factor Hfq, as the bacterial branch of the Sm family, is an RNA-binding protein involved in the post-transcriptional regulation of mRNA expression and turnover. Hfq facilitates pairing between small regulatory RNAs (sRNAs) and their corresponding mRNA targets by binding both RNAs and bringing them into close proximity. Hfq homologs self-assemble into homo-hexameric rings with at least two distinct surfaces that bind RNA. Recently, another binding site, dubbed the `lateral rim', has been implicated in sRNA·mRNA annealing; the RNA-binding properties of this site appear to be rather subtle, and its degree of evolutionary conservation is unknown. An Hfq homolog has been identified in the phylogenetically deep-branching thermophile Aquifex aeolicus ( Aae ), but little is known about the structure and function of Hfq from basal bacterial lineages such as the Aquificae. Therefore, Aae Hfq was cloned, overexpressed, purified, crystallized and biochemically characterized. Structures of Aae Hfq were determined in space groups P 1 and P 6, both to 1.5 Å resolution, and nanomolar-scale binding affinities for uridine- and adenosine-rich RNAs were discovered. Co-crystallization with U 6 RNA reveals that the outer rim of the Aae Hfq hexamer features a well defined binding pocket that is selective for uracil. This Aae Hfq structure, combined with biochemical and biophysical characterization of the homolog, reveals deep evolutionary conservation of the lateral RNA-binding mode, and lays a foundation for further studies of Hfq-associated RNA biology in ancient bacterial phyla.

59 BASIC BIOLOGICAL SCIENCES↗

Energy Efficient Stochastic-Based Deep Spiking Neural Networks for Sparse Datasets

With large deep neural networks (DNNs) neces- sary to solve complex and data-intensive problems, energy efficiency is a key bottleneck for effectively deploying DL in the real world. Deep spiking NNs have gained much research at- tention recently due to the interest in building biological neural networks and the availability of neuromorphic platforms, which can be orders of magnitude more energy efficient compared to CPUs and GPUs. Although spiking NNs have proven to be an efficient technique for solving many machine learning and computer vision problems, to the best of our knowledge, this is the first attempt to adapt spiking NNs to sparse datasets. In this paper, we study the behaviour of spiking NNs in handling NLP datasets and the sparsity in their data representation. Then, we propose a novel framework for spiking NN using the concept of stochastic computing. Specifically, instead of generating spike trains with firing rates proportional to the intensity of each value in the feature set separately, the whole feature set is treated as a distribution function and a stochastic spiking train that follow this distribution is generated. This framework reduces the connectivity between NN layers from O(N) to O(logN). Also, it encodes input data differently and make suitable to handle sparse datasets. Finally, the framework achieves high energy efficiency since it uses Integrate and Fire neurons same as conventional spiking NNs. The results show that our proposed stochastic-based SNN achieves nearly the same accuracy as the original DNN on MNIST dataset, and it has better performance than state- of-the-art SNN. Besides that stochastic-based SNN is energy efficient, where the fully connected DNN, the conventional SNN, and the data normalized SNN consume 38.24, 1.83, and 1.85 -times more energy than the stochastic-based SNN, respectively. For sparse datasets, including IMDb and In-House clinical datasets, stochastic-based SNN achieves performance comparable to that of the conventional DNN. However, the conventional spiking NN has a significant decline in classification accuracy.

Alawad, Mohammed↗

Adversarial Training for Privacy-Preserving Deep Learning Model Distribution

Collaboration among cancer registries is essential to develop accurate, robust, and generalizable deep learning models for automated information extraction from cancer pathology reports. Sharing data presents a serious privacy issue, especially in biomedical research and healthcare delivery domains. Distributing pretrained deep learning (DL) models has been proposed to avoid critical data sharing. However, there is growing recognition that collaboration among clinical institutes through DL model distribution exposes new security and privacy vulnerabilities. These vulnerabilities increase in natural language processing (NLP) applications, in which the dataset vocabulary with word vector representations needs to be associated with the other model parameters. In this paper, we propose a novel privacy-preserving DL model distribution across cancer registries for information extraction from cancer pathology reports with privacy and confidentiality considerations. The proposed approach exploits the adversarial training framework to distinguish private features from shared features among different datasets. It only shares registry-invariant model parameters, without sharing raw data nor registry-specific model parameters among cancer registries. Thus, it protects both the data and the trained model simultaneously. We compare our proposed approach to single-registry models, and a model trained on centrally hosted data from different cancer registries. The results show that the proposed approach significantly outperforms the single-registry models and achieves statistically indistinguishable micro and macro F1-score as compared to the centralized model.

Alawad, Mohammed↗

A Deep Learning Pipeline for Optimizing Large-scale Phase Field Simulations

Phase field (PF) simulations are computationally expensive but remain a key analysis tool to understand the complex mechanisms of additive manufacturing (AM) processes. Each PF simulation-aided analysis requires thousands of node hours on leadership-class supercomputers. One of the main goals of these analyses is the study of microstructure evolution during the build process which begins with the onset of nucleation. Nucleation occurs under certain thermomechanical conditions which are not known a priori and many PF simulations are required to identify ranges of input thermo-mechanical parameters that can result in the onset of nucleation. Since many of the simulations do not result in nucleation, an analysis campaign often ends up wasting tremendous amounts of precious computing resources executing nucleation-absent simulations. The goal of this work is to design and train deep learning models to inform a PF simulation about the likelihood of the occurrence of nucleation in a future simulation time-step based on the state summary over a finite number of past time-steps of a running simulation. If the prediction determines that the running simulation is unlikely to reach nucleation in the allotted time, then its execution is stopped immediately ultimately resulting in vast reduction in wasted computations when accrued over all the PF simulations typically performed in a single or multiple analysis campaign(s). The paper presents the performance of a machine learning pipeline that uses a convolutional neural network (CNN) model to learn an embedding which is then used with a self-attention network to build a multi-task deep learning model to predict the likelihood of nucleation. The model also predicts the input parameters used in a simulation. Performance is compared with a baseline pipeline that uses an off-the-shelf LeNet-5 model to learn the initial embedding. Despite their smaller size, performance results indicate significant improvement in accuracy of the proposed models compared to the larger baseline models.

Kannan, Ramakrishnan {ramki}↗

Data optimization for large batch distributed training of deep neural networks

Distributed training in deep learning (DL) is common practice as data and models grow. The current practice for distributed training of deep neural networks faces the challenges of communication bottlenecks when operating at scale, and model accuracy deterioration with an increase in global batch size. Present solutions focus on improving message exchange efficiency as well as implementing techniques to tweak batch sizes and models in the training process. The loss of training accuracy typically happens because the loss function gets trapped in a local minima. We observe that the loss landscape minimization is shaped by both the model and training data and propose a data optimization approach that utilizes machine learning to implicitly smooth out the loss landscape resulting in fewer local minima. Our approach filters out data points which are less important to feature learning, enabling us to speed up the training of models on larger batch sizes to improved accuracy.

Gahlot, Shubhankar↗

Evolving Larger Convolutional Layer Kernel Sizes for a Settlement Detection Deep-Learner on Summit

Deep-learner hyper-parameters, such as kernel sizes, batch sizes, and learning rates, can significantly influence the quality of trained models. The state of the art for finding optimal hyper-parameters generally uses a brute force, grid search approach, random search, or Bayesian-based optimization among other techniques. We applied an evolutionary algorithm to optimize kernel sizes for a convolutional neural network used to detect settlements in satellite imagery. Usually convolutional layer kernel sizes are small - typically one, three, or five - but we found that the system converged at, or near, kernel sizes of nine for the last convolutional layer, and that this occurred for multiple runs using two different datasets. Moreover, the larger kernel sizes had fewer false positives than the 3x3 kernel sizes found as optimal via a brute force uniform grid search. This suggests that this large kernel size may be leveraging patterns found in larger areal features in the source imagery, and that this may be generalized as possible guidance for similar remote sensing deep-learning tasks.

Coletti, Mark↗

Optimal Coordination of Distributed Energy Resources Using Deep Deterministic Policy Gradient

Recent studies showed that reinforcement learning (RL) is a promising approach for coordination and control of distributed energy resources (DER) under uncertainties. Many existing RL approaches, including Q-learning and approximate dynamic programming, are based on lookup table methods, which become inefficient when the problem size is large and infeasible when continuous states and actions are involved. In addition, when modeling battery energy storage system (BESS), the loss of life is not reasonably considered into the decision-making process. This paper proposes an innovative deep RL method for DER coordination considering BESS degradation. The proposed deep RL is designed based on an adaptive actor-critic architecture and employs an off-policy deterministic policy gradient method for determining the dispatch operation that minimizes the operation cost and BESS life loss. Case studies were performed to validate the proposed method and demonstrate the effects of incorporating degradation models into control design.

Das, Avijit↗

Entropy-Infused Deep Learning Loss Function for Capturing Extreme Values in Wind Power Forecasting

Extreme scenarios in wind power generation occur with higher frequency and larger magnitude in the recent years due to the ever-increasing extreme meteorological factors. Accurate forecasting of the occurrence of extreme values in wind power generation is of great concern to ensure reliable power system operation. Recently, deep learning models have surged in popularity for wind power forecasting, with the mean squared error (MSE) loss function being commonly used. However, the MSE loss function, being sensitive to extreme values, disproportionately penalizes larger errors, cannot adequately capture the extreme values present in wind energy data, and novel loss functions have seldom been tailored for wind power forecasting. To this end, in this paper, we introduce a novel loss function specifically crafted to capture extreme values in wind power forecasting. The experimental results with four fundamental deep learning methods on open source wind power dataset validate that the new loss function is efficient and superior in all cases compared to MSE in capturing extreme values while maintaining forecasting performance.

17 WIND ENERGY↗

Distributed-Memory Sparse Deep Neural Network Inference Using Global Arrays

Partitioned Global Address Space (PGAS) models exhibit tremendous promise in developing efficient and productive distributed-memory parallel applications. They have been used extensively in scientific computations due to conveniently offering a ``shared-memory''-like model and convenient interfaces that separate communication with synchronization. Traditionally, PGAS communication models have been applied to dense/contiguously distributed data, but most modern applications depict varied levels of sparsity. Existing PGAS models require certain adaptations to support distributed sparse computations, since associated computations often require matrix arithmetic, in addition to data movement. The Global Arrays toolkit from Pacific Northwest National Laboratory (PNNL) is one of the earliest PGAS models to combine one-sided data communication and distributed matrix operations and is still used in the popular NWChem quantum chemistry suite. Recently, we have expanded the Global Arrays toolkit to support common sparse operations, like sparse matrix-dense matrix multiplies (SpMM), sparse matrix-sparse matrix multiplication (SpGEMM) and Sampled Dense-Dense Matrix Multiplication (SDDMM). As it turns out, these operations are the bedrock of sparse Deep Learning (DL); sparse deep neural networks and Graph Neural Networks (GNNs) have gained increasing attention recently in achieving speedups on training and inference with reduced memory footprints. Unlike scientific applications in High Performance Computing (HPC), modern (distributed-memory capable) DL toolkits often rely on non-standardized and closed-source vendor software optimizations, creating challenges in software-hardware co-design at scale. Our goal is to support a variety of distributed-memory sparse matrix operations and helper functions in the newly created Sparse Global Arrays (SGA), such that it is possible to build portable and productive Machine Learning scenarios for algorithm/software and hardware codesign purposes. Contemporary data-parallel schemes for training/inference are undergoing a major overhaul since model replication limits scalability and causes resource inefficiencies. As such, we have adopted tensor parallelism in decomposing the model and inputs, to mitigate memory issues. Current implementation is built on top of MPI and uses CPUs to maximize the portability across the platforms.

Distributed computing, machine learning↗

Deep Transfer Learning With Field-Based Measurements for Large Area Classification

Classification of Electro-Optical (EO) datasets using Deep Neural Networks (DNNs) has lead to high-performance supervised learning algorithms. However, DNNs require a large amount of labeled data for training which often have limited availability in ecological studies. Up-to-date classification of vegetation in sensitive Arctic ecosystems continue to be a challenge. In an ecosystem undergoing rapid change, capturing the dynamics of vegetation requires the existing maps of vegetation (such as Alaska Existing Vegetation Type (AKEVT), circa 2000) to be updated based on frequent field based observation of vegetation. A method is needed to transfer the knowledge gained from field-collected observations to a larger area using DNNs. Transfer learning is a machine learning technique where a model trained on one task is re-purposed on another similar task. This paper seeks to train DNNs using field-collected observations and apply transfer learning to apply the knowledge gained to update existing vegetation map at larger scale. We test two DNN methods, (1) a deep multilayer perceptron (MLP) model and (2) siamese MLP network that uses a structure to rank similarity between inputs and can be used for training datasets with few samples and show good performance with limited datasets (e.g. few-shot learning). The results show ∼90% accuracy (using the field observations for evaluation) when transfer learning is applied to a siamese network, compared to ∼45% accurate when a MLP is trained on the AKEVT and evaluated on the field observations. The approach show promise for improving and update the existing vegetation maps over large areas using limited field-based observations.

Langford, Zach↗

Revisit the Scalability of Deep Auto-Regressive Models for Graph Generation

As a new promising approach to graph generations, deep auto-regressive graph generation has drawn increasing attention. It however has been commonly deemed as hard to scale up to work with large graphs. In existing studies, it is perceived that the consideration of the full non-local graph dependences is indispensable for this approach to work, which entails the needs for keeping the entire graph’s info in memory and hence the perceived “inherent” scalability limitation of the approach. This paper revisits the common perception. It proposes three ways to relax the dependences and conducts a series of empirical measurements. It concludes that the perceived “inherent” scalability limitation is a misperception; with the right design and implementation, deep auto-regressive graph generation can be applied to graphs much larger than the device memory. The rectified perception removes a fundamental barrier for this approach to meet practical needs.

Yang, Shuai↗

DeepGrid: Robust Deep Reinforcement Learning-based Contingency Management

Increasing uncertainty raised by the integration of renewable energy resources requires an enormous number of simulations to be carried out for the security assessment of the power grid. However, it is challenging to assess the steady-state and dynamic security indices for different system contingency events by doing an exhaustive analysis in real-time due to the computational and communication constraints. One promising solution is using data-driven techniques along with the system models to train an intelligent contingency management framework to better handle the contingencies in real-time. Nevertheless, implementing a data-driven technique to obtain the best remedial actions necessitates to account for the effect of the measurement noise on the performance of the contingency management. To tackle these challenges, we leverage a robust deep reinforcement learning (DRL) algorithm called Double Deep Q-Network (DDQN) to design a recommender system capable of prescribing optimal control actions with the help of the real-time digital simulator (RTDS). The use of RTDS system in combination with the advanced DRL algorithm allows to explore a wide variety of system contingencies in order to derive better remedial actions. The performance of the proposed algorithm is evaluated in IEEE 9-bus system under different loading conditions, and different network configurations in presence of noisy measurements.

Ghasemkhani, Amir↗

DeepGraphONet: A Deep Graph Operator Network to Learn and Zero-Shot Transfer the Dynamic Response of Networked Systems

This article develops a deep graph operator network (DeepGraphONet) framework that learns to approximate the dynamics of a complex system (e.g., the power grid or traffic) with an underlying subgraph structure. Here, we build our DeepGraphONet by fusing the ability of graph neural networks to exploit spatially correlated graph information and deep operator networks to approximate the solution operator of dynamical systems. The resulting DeepGraphONet can then predict the dynamics within a given short/medium-term time horizon by observing a finite history of the graph state information. Furthermore, we design our DeepGraphONet to be resolution independent. That is, we do not require the finite history to be collected at the exact/same resolution. In addition, to disseminate the results from a trained DeepGraphONet, we design a zero-shot learning strategy that enables using it on a different subgraph. Finally, empirical results on the transient stability prediction problem of power grids and traffic flow forecasting problem of a vehicular system illustrate the effectiveness of the proposed DeepGraphONet.

24 POWER TRANSMISSION AND DISTRIBUTION↗