Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “deep learning (DL)”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Distributed-Memory Sparse Deep Neural Network Inference Using Global Arrays

Partitioned Global Address Space (PGAS) models exhibit tremendous promise in developing efficient and productive distributed-memory parallel applications. They have been used extensively in scientific computations due to conveniently offering a ``shared-memory''-like model and convenient interfaces that separate communication with synchronization. Traditionally, PGAS communication models have been applied to dense/contiguously distributed data, but most modern applications depict varied levels of sparsity. Existing PGAS models require certain adaptations to support distributed sparse computations, since associated computations often require matrix arithmetic, in addition to data movement. The Global Arrays toolkit from Pacific Northwest National Laboratory (PNNL) is one of the earliest PGAS models to combine one-sided data communication and distributed matrix operations and is still used in the popular NWChem quantum chemistry suite. Recently, we have expanded the Global Arrays toolkit to support common sparse operations, like sparse matrix-dense matrix multiplies (SpMM), sparse matrix-sparse matrix multiplication (SpGEMM) and Sampled Dense-Dense Matrix Multiplication (SDDMM). As it turns out, these operations are the bedrock of sparse Deep Learning (DL); sparse deep neural networks and Graph Neural Networks (GNNs) have gained increasing attention recently in achieving speedups on training and inference with reduced memory footprints. Unlike scientific applications in High Performance Computing (HPC), modern (distributed-memory capable) DL toolkits often rely on non-standardized and closed-source vendor software optimizations, creating challenges in software-hardware co-design at scale. Our goal is to support a variety of distributed-memory sparse matrix operations and helper functions in the newly created Sparse Global Arrays (SGA), such that it is possible to build portable and productive Machine Learning scenarios for algorithm/software and hardware codesign purposes. Contemporary data-parallel schemes for training/inference are undergoing a major overhaul since model replication limits scalability and causes resource inefficiencies. As such, we have adopted tensor parallelism in decomposing the model and inputs, to mitigate memory issues. Current implementation is built on top of MPI and uses CPUs to maximize the portability across the platforms.

Distributed computing, machine learning↗

TNet: A Model-Constrained Tikhonov Network Approach for Inverse Problems

Deep learning (DL), in particular deep neural networks, by default is purely data-driven and in general does not require physics. This is the strength of DL but also one of its key limitations when applied to science and engineering problems in which underlying physical properties—such as stability, conservation, and positivity—and accuracy are required. DL methods in their original forms are often not capable of respecting the underlying mathematical models or achieving desired accuracy even in big-data regimes. On the other hand, many data-driven science and engineering problems, such as inverse problems, typically have limited experimental or observational data, and DL would overfit the data in this case. Leveraging information encoded in the underlying mathematical models, we argue, not only compensates for missing information in low data regimes but also provides opportunities to equip DL methods with the underlying physics, hence promoting better generalization. This paper develops a model-constrained DL approach and its variant TNet—a Tikhonov neural network—which are capable of learning not only information hidden in the training data but also in the underlying mathematical models to solve inverse problems governed by partial differential equations in low data regimes. We provide the constructions and some theoretical results for the proposed approaches for both linear and nonlinear inverse problems. Since TNet is designed to learn inverse solutions with Tikhonov regularization, it is interpretable: in fact it recovers Tikhonov solutions for linear cases while potentially approximating Tikhonov solutions for nonlinear inverse problems. We also prove that data randomization can enhance not only the smoothness of the networks but also their generalizations. Comprehensive numerical results confirm the theoretical findings and show that with even as little as 1 training data sample for one-dimensional (1D) deconvolution, 5 for an inverse 2D heat conductivity problem, 100 for inverse initial conditions for a time-dependent 2D Burgers’s equation, and 50 for inverse initial conditions for 2D Navier–Stokes equations, TNet solutions can be as accurate as Tikhonov solutions while being several orders of magnitude faster. Furthermore, this is possible owing to the model-constrained term, replications, and randomization.

97 MATHEMATICS AND COMPUTING↗

DeepHyper: Asynchronous Hyperparameter Search for Deep Neural Networks

Hyperparameters employed by deep learning (DL) methods play a substantial role in the performance and reliability of these methods in practice. Unfortunately, finding performance-optimizing hyperparameter settings is a notoriously difficult task. Hyperparameter search methods typically have limited production-strength implementations or do not target scalability within a highly parallel machine, portability across different machines, experimental comparison between different methods, and tighter integration with workflow systems. In this paper, we present DeepHyper, a Python package that provides a common interface for the implementation and study of scalable hyperparameter search methods. It adopts the Balsam workflow system to hide the complexities of running large numbers of hyperparameter configurations in parallel on high-performance computing (HPC) systems. We implement and study asynchronous model-based search methods that consist of sampling a small number of input hyperparameter configurations and progressively fitting surrogate models over the input-output space until exhausting a user-defined budget of evaluations. We evaluate the efficacy of these methods relative to approaches such as random search, genetic algorithms, Bayesian optimization, and hyperband on DL benchmarks on CPU- and GPU-based HPC systems.

Balaprakash, Prasanna↗

Scaling deep learning workloads: NVIDIA DGX-1/Pascal and Intel Knights Landing

Deep Learning (DL) algorithms have become ubiquitous in data analytics. As a result, major computing vendors --- including NVIDIA, Intel, AMD, and IBM --- have architectural road-maps influenced by DL workloads. Furthermore, several vendors have recently advertised new computing products as accelerating large DL workloads. Unfortunately, it is difficult for data scientists to quantify the potential of these different products. This paper provides a performance and power analysis of important DL workloads on two major parallel architectures: NVIDIA DGX-1 (eight Pascal P100 GPUs interconnected with NVLink) and Intel Knights Landing (KNL) CPUs interconnected with Intel Omni-Path or Cray Aries. Our evaluation consists of a cross section of convolutional neural net workloads: CifarNet, AlexNet, GoogLeNet, and ResNet50 topologies using the Cifar10 and ImageNet datasets. The workloads are vendor-optimized for each architecture. Our analysis indicates that although GPUs provide the highest overall performance, the gap can close for some convolutional networks; and the KNL can be competitive in performance/watt. We find that NVLink facilitates scaling efficiency on GPUs. However, its importance is heavily dependent on neural network architecture. Furthermore, for weak-scaling --- sometimes encouraged by restricted GPU memory --- NVLink is less important.

Gawande, Nitin A.↗

Deep inverse photonic design: A tutorial

Deep learning (DL) has been used to design deep neural networks (DNNs) which have recently been applied to solving inverse problems in artificial electromagnetic materials (AEMs). Although inverse problems are often ill-posed, and therefore are difficult to solve, deep inverse models (DIMs) have achieved impressive results often surpassing capabilities possible with other approaches. We overview the process of deep inverse learning applied to AEM problems, including the building of data sets, design of a forward model, and comparison of inverse approaches including limitations. We conclude by detailing some important outstanding issues of deep inverse design of AEMs, and present an outlook of this exciting field.

42 ENGINEERING↗

Audiovisual Fusion: Challenges and New Approaches

In this study, we review recent results on audiovisual (AV) fusion. We also discuss some of the challenges and report on approaches to address them. One important issue in AV fusion is how the modalities interact and influence each other. This review will address this question in the context of AV speech processing, and especially speech recognition, where one of the issues is that the modalities both interact but also sometimes appear to desynchronize from each other. An additional issue that sometimes arises is that one of the modalities may be missing at test time, although it is available at training time; for example, it may be possible to collect AV training data while only having access to audio at test time. We will review approaches to address this issue from the area of multiview learning, where the goal is to learn a model or representation for each of the modalities separately while taking advantage of the rich multimodal training data. In addition to multiview learning, we also discuss the recent application of deep learning (DL) toward AV fusion. We finally draw conclusions and offer our assessment of the future in the area of AV fusion.

47 OTHER INSTRUMENTATION↗

The Application of Artificial Intelligence and Deep Learning to Visually Identify Micrometeoroid and Orbital Debris Impacts

Recent advancements in Artificial Intelligence (AI) have made Machine Learning (ML) techniques readily available for practical applications while using a fraction of time that was previously required. In particular, the use of Deep Learning (DL) algorithms has advanced the field of image and pattern recognition. With the use of Deep Learning algorithms, Micrometeoroid and Orbital Debris(MMOD) penetrations can be identified with high accuracy and give possibilities to new understandings of hypervelocity impacts.

Deep Learning↗

A Study of Physics-Informed Deep Learning for System Fluid Dynamics Closures

The sub-grid-scale (SGS) physics models, or so-called closure relations (CR), are essential in thermal-fluid modeling and simulation codes, ranging from large-eddy simulation to system-level thermo-fluid dynamic problems. While details vary, all thermo-fluid codes solve coupled mass-momentum-energy conservation partial differential equations (PDE). Traditionally, the analytical form of closure models has to be known, and then we calibrate the model using data from a set of applicable experiments. Characteristically, it takes long effort to gain insights and mechanistic understanding needed for developing closure models. Such a model is a compact form of data. The long time required for new model development limits the pace of model application while dealing with new geometries, new coolants or system conditions, particularly in newly designed systems. Alternative to mechanistic and semi-analytical models, it is possible to discover the pattern of closure relations by applying the modern machine learning methods to the body of relevant experimental data. Deep learning (DL) is a relatively new branch in the machine learning family, and a major breakthrough has been achieved since Hinton introduces a fast algorithm to efficiently train a feed-forward neural network (NN) with multilayer perceptrons. Generally, any NN with more than two layers is considered as deep learning, and it contains a large number of hyper-parameters to achieve pattern recognition or regression for complex datasets. Hornik shows that multilayer NN is compatible with universal approximation theorem. There is no constraint for a multilayer NN to capture the properties of any measurable information. This capability makes DL attractive for closure models in a system thermal-hydraulic code. As system geometry and/or operating conditions change, new physical experiments (increasingly sophisticated) are conducted to support evaluation and revision of closure models. Such a so-called data-driven modeling relaxes the structural form of system codes to directly assimilate data instead of hard-wiring the model to pre-calibrated correlations. The objective of this research is to enable cost-effective development of a DL-based closure that reflects the data. The previous work on applying machine learning techniques to fluid dynamics focused on statistical learning with basic NN from direct numerical simulations (DNS) dataset. In the present work, we show that DL-based closure models can cause the fluid calculation to diverge if multilayer NN is used. To address the issue, we investigate conditions by which the DL-based closure models work compatibly, stably, and effectively with PDE-constrained forward prediction problems. Optimally meeting the compatibility conditions constitutes a physics-informed deep learning (PIDL) strategy. We refer to this process as PDE-constrained machine learning or PDE-ML. To develop the technical basis for PDE-ML using DL-based fluid dynamics closure, numerical experiments are performed for manufactured problems of increasing complexity. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Deep Learning Based Superconducting Radio-Frequency Cavity Fault Classification at Jefferson Laboratory

This work investigates the efficacy of deep learning (DL) for classifying C100 superconducting radio-frequency (SRF) cavity faults in the Continuous Electron Beam Accelerator Facility (CEBAF) at Jefferson Lab. CEBAF is a large, high-power continuous wave recirculating linac that utilizes 418 SRF cavities to accelerate electrons up to 12 GeV. Recent upgrades to CEBAF include installation of 11 new cryomodules (88 cavities) equipped with a low-level RF system that records RF time-series data from each cavity at the onset of an RF failure. Typically, subject matter experts (SME) analyze this data to determine the fault type and identify the cavity of origin. This information is subsequently utilized to identify failure trends and to implement corrective measures on the offending cavity. Manual inspection of large-scale, time-series data, generated by frequent system failures is tedious and time consuming, and thereby motivates the use of machine learning (ML) to automate the task. This study extends work on a previously developed system based on traditional ML methods (Tennant and Carpenter and Powers and Shabalina Solopova and Vidyaratne and Iftekharuddin, Phys. Rev. Accel. Beams, 2020, 23, 114601), and investigates the effectiveness of deep learning approaches. The transition to a DL model is driven by the goal of developing a system with sufficiently fast inference that it could be used to predict a fault event and take actionable information before the onset (on the order of a few hundred milliseconds). Because features are learned, rather than explicitly computed, DL offers a potential advantage over traditional ML. Specifically, two seminal DL architecture types are explored: deep recurrent neural networks (RNN) and deep convolutional neural networks (CNN). We provide a detailed analysis on the performance of individual models using an RF waveform dataset built from past operational runs of CEBAF. In particular, the performance of RNN models incorporating long short-term memory (LSTM) are analyzed along with the CNN performance. Furthermore, comparing these DL models with a state-of-the-art fault ML model shows that DL architectures obtain similar performance for cavity identification, do not perform quite as well for fault classification, but provide an advantage in inference speed.

97 MATHEMATICS AND COMPUTING↗

Improving and Assessing the Quality of Uncertainty Quantification in Deep Learning

Deep learning (DL) models have enjoyed increased attention in recent years because of their powerful predictive capabilities. While many successes have been achieved, standard deep learning methods suffer from a lack of uncertainty quantification (UQ). While the development of methods for producing UQ from DL models is an active area of current research, little attention has been given to the quality of the UQ produced by such methods. In order to deploy DL models to high-consequence applications, high-quality UQ is necessary. This report details the research and development conducted as part of a Laboratory Directed Research and Development (LDRD) project at Sandia National Laboratories. The focus of this project is to develop a framework of methods and metrics for the principled assessment of UQ quality in DL models. This report presents an overview of UQ quality assessment in traditional statistical modeling and describes why this approach is difficult to apply in DL contexts. An assessment on relatively simple simulated data is presented to demonstrate that UQ quality can differ greatly between DL models trained on the same data. A method for simulating image data that can then be used for UQ quality assessment is described. A general method for simulating realistic data for the purpose of assessing a model’s UQ quality is also presented. A Bayesian uncertainty framework for understanding uncertainty and existing metrics is described. Research that came out of collaborations with two university partners are discussed along with a software toolkit that is currently being developed to implement the UQ quality assessment framework as well as serve as a general guide to incorporating UQ into DL applications.

97 MATHEMATICS AND COMPUTING↗

Deep learning inversion of gravity data for detection of CO 2 plumes in overlying aquifers

In this work, we developed an effective U-Net based deep learning (DL) model for inversion of surface gravity data on a rectangular grid to predict 2-D high-resolution subsurface CO 2 distribution along a vertical cross-section due to CO 2 leakage through a wellbore within a deep CO 2 storage reservoir. We used synthetic data to model two types of CO 2 leakage scenarios: one CO 2 plume in a shallow aquifer (single plume case), and two plumes present at different depths (double plume case). The 3-D synthetic plume samples were created by sampling among predetermined CO 2 plume depths, saturations, and volumes. The corresponding surface gravity data on a rectangular grid were generated by a 3-D forward model. The U-Net model detected 72% of single-plume samples, and one or both plumes in 75% of double-plume samples. Most of the undetected single plumes have small gravity field strengths below the typical noise level of 5 μGal. This model generated reproducible, reliable predictions with acceptable errors and demonstrated improved spatial resolution over the conventional least-squares inversion. In contrast to the conventional least-squares inversion, which often overestimates the size of its target and underestimates its density, this U-Net model accurately delineated the boundary of a target. Furthermore, this DL inversion detected deep, small, or low saturation CO 2 plumes that are often more difficult to resolve with conventional gravity inversion methods. We note the limitations of this feasibility study, including the use of synthetic data with regular CO 2 plume shapes, and the prediction of a 2-D plume cross-section rather than the full 3-D plume, as well, we recognize the lower detection fraction for double-plume scenarios. Nevertheless, this study demonstrates that DL gravity inversion is a promising and potentially superior method to conventional least-squares inversion. Our U-Net based deep learning inversion approach may be adapted for inversion of other types of geophysical data. DL inversion can facilitate near real-time monitoring of geologic carbon sequestration to provide site operators with prompt information about subsurface CO 2 distribution for risk management and mitigation.

58 GEOSCIENCES↗

Deep learning surrogate model for kinetic Landau-fluid closure with collision

In this work, the kinetic Landau-fluid (LF) closure with collision and periodic boundary condition is used in the development of the deep learning (DL) surrogate model. A classical neural network, namely, feedforward neural network or sometimes termed multilayer perceptron, is constructed and trained to learn the kinetic LF closure in the static limit and arbitrary mean free path in configuration space. The preliminary relation between best hyperparameters and critical parameters for data generation is found. Compared with the numerical approach (non-Fourier method) of the LF closure, the deep learning surrogate model shows an order of magnitude of improvement in terms of accuracy. Perhaps most importantly, the surrogate model closure has been integrated for the first time with fluid simulations. Our DL-enabled fluid simulations, for the first time, give the correct Landau damping rate for a wide range of wave vectors, while the Hammett–Perkins closure cannot produce the correct damping rate. We correctly connect the collisionless Hammett–Perkins closure and collisional Braginskii closure to reproduce the intrinsic nonlocal feature of the heat flux with DL techniques. We address the most concerning error accumulation problem and find that simulations with the deep learning surrogate model are as good as, if not better than, simulations with the analytic closure in terms of long-term numerical stability in the linear Landau damping test.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

User-level File Systems Specialized for HPC Workloads Year-End Report

High-performance computing (HPC) clusters attract the attention of Deep Learning (DL) training users due to the clusters' powerful computation capabilities. While there have been many existing efforts to enable deep neural works to leverage the powerful CPU and GPU processors from leadership high-performance computing (HPC) systems, large-scale deep learning with larger datasets requires efficient I/O support from the underlying file and storage systems. As some current and upcoming HPC clusters have large on-node memory or are equipped with NVMe SSD on compute nodes, more distributed DL trainings are considered to leverage those storages to store dataset for efficient dataset access. Our project goal is to design a specialized DL-oriented file system to improve datasets loading performance of any DL training applications. Over the past year, the team at Florida State University has performed research activities in three different aspects, including 1) completing a specialized memory-based I/O framework (DeepIO) for improving dataset loading performance on DL applications; 2) proposing a more generalized file system (DLFS) for DL applications on node-local SSDs; 3) completing the BeeGFS performance evaluation project for Deep Neural Networks.

97 MATHEMATICS AND COMPUTING↗

Recent advances and applications of deep learning methods in materials science

Deep learning (DL) is one of the fastest-growing topics in materials data science, with rapidly emerging applications spanning atomistic, image-based, spectral, and textual data modalities. DL allows analysis of unstructured data and automated identification of features. The recent development of large materials databases has fueled the application of DL methods in atomistic prediction in particular. In contrast, advances in image and spectral data have largely leveraged synthetic data enabled by high-quality forward models as well as by generative unsupervised DL methods. In this article, we present a high-level overview of deep learning methods followed by a detailed discussion of recent developments of deep learning in atomistic simulation, materials imaging, spectral analysis, and natural language processing. For each modality we discuss applications involving both theoretical and experimental data, typical modeling approaches with their strengths and limitations, and relevant publicly available software and datasets. We conclude the review with a discussion of recent cross-cutting work related to uncertainty quantification in this field and a brief perspective on limitations, challenges, and potential growth areas for DL methods in materials science.

36 MATERIALS SCIENCE↗

Analyzing inference workloads for spatiotemporal modeling

Ensuring power grid resiliency, forecasting climate conditions, and optimization of transportation infrastructure are some of the many application areas where data is collected in both space and time. Spatiotemporal modeling is about modeling those patterns for forecasting future trends and carrying out critical decision-making by leveraging machine learning/deep learning. Once trained offline, field deployment of trained models for near real-time inference could be challenging because performance can vary significantly depending on the environment, available compute resources and tolerance to ambiguity in results. Users deploying spatiotemporal models for solving complex problems can benefit from analytical studies considering a plethora of system adaptations to understand the associated performance-quality trade-offs. To facilitate the co-design of next-generation hardware architectures for field deployment of trained models, it is critical to characterize the workloads of these deep learning (DL) applications during inference and assess their computational patterns at different levels of the execution stack. In this paper, we develop several variants of deep learning applications that use spatiotemporal data from dynamical systems. We study the associated computational patterns for inference workloads at different levels, considering relevant models (Long short-term Memory, Convolutional Neural Network and Spatio-Temporal Graph Convolution Network), DL frameworks (Tensorflow and PyTorch), precision (FP16, FP32, AMP, INT16 and INT8), inference runtime (ONNX and AI Template), post-training quantization (TensorRT) and platforms (Nvidia DGX A100 and Sambanova SN10 RDU). Overall, our findings indicate that although there is potential in mixed-precision models and post-training quantization for spatiotemporal modeling, extracting efficiency from contemporary GPU systems might be challenging. Instead, co-designing custom accelerators by leveraging optimized High Level Synthesis frameworks (such as SODA High-Level Synthesizer for customized FPGA/ASIC targets) can make workload-specific adjustments to enhance the efficiency.

97 MATHEMATICS AND COMPUTING↗

Surrogate Modeling of Nonlinear Dynamic Systems: A Comparative Study

Surrogate models play a vital role in overcoming the computational challenge in designing and analyzing nonlinear dynamic systems, especially in the presence of uncertainty. This paper presents a comparative study of different surrogate modeling techniques for nonlinear dynamic systems. Four surrogate modeling methods, namely, Gaussian process (GP) regression, a long short-term memory (LSTM) network, a convolutional neural network (CNN) with LSTM (CNN-LSTM), and a CNN with bidirectional LSTM (CNN-BLSTM), are studied and compared. All these model types can predict the future behavior of dynamic systems over long periods based on training data from relatively short periods. The multi-dimensional inputs of surrogate models are organized in a nonlinear autoregressive exogenous model (NARX) scheme to enable recursive prediction over long periods, where current predictions replace inputs from the previous time window. Three numerical examples, including one mathematical example and two nonlinear engineering analysis models, are used to compare the performance of the four surrogate modeling techniques. The results show that the GP-NARX surrogate model tends to have more stable performance than the other three deep learning (DL)-based methods for the three particular examples studied. The tuning effort of GP-NARX is also much lower than its deep learning-based counterparts.

42 ENGINEERING↗

Data optimization for large batch distributed training of deep neural networks

Distributed training in deep learning (DL) is common practice as data and models grow. The current practice for distributed training of deep neural networks faces the challenges of communication bottlenecks when operating at scale, and model accuracy deterioration with an increase in global batch size. Present solutions focus on improving message exchange efficiency as well as implementing techniques to tweak batch sizes and models in the training process. The loss of training accuracy typically happens because the loss function gets trapped in a local minima. We observe that the loss landscape minimization is shaped by both the model and training data and propose a data optimization approach that utilizes machine learning to implicitly smooth out the loss landscape resulting in fewer local minima. Our approach filters out data points which are less important to feature learning, enabling us to speed up the training of models on larger batch sizes to improved accuracy.

Gahlot, Shubhankar↗

A Public Data Set of Auto-Generated Geotagged PV Site Equipment, Generated via Deep Learning

In this research, we present a data set over 100 photovoltaic (PV) sites in TX, which have been automatically geotagged via a fully autonomous deep learning (DL) pipeline. Specifically, locations of inverters, tracker/fixed tilt rows, batteries, and substations are labeled algorithmically. To ensure high data quality, all systems have been reviewed manually and any deep learning errors have been corrected. This public data set, as well as the open-sourced pipeline used to generate it, is valuable for site planning, modelling, and insurance purposes. Given time and resources, we hope to extend the data set to additional states/regions in the US.

14 SOLAR ENERGY↗