SEARCH · Engineering Papers

Results for “dl”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22

All sources NASA NTRS DOE OSTI

DOE OSTI2017

Energy Efficient Stochastic-Based Deep Spiking Neural Networks for Sparse Datasets

With large deep neural networks (DNNs) neces- sary to solve complex and data-intensive problems, energy efficiency is a key bottleneck for effectively deploying DL in the real world. Deep spiking NNs have gained much research at- tention recently due to the interest in building biological neural networks and the availability of neuromorphic platforms, which can be orders of magnitude more energy efficient compared to CPUs and GPUs. Although spiking NNs have proven to be an efficient technique for solving many machine learning and computer vision problems, to the best of our knowledge, this is the first attempt to adapt spiking NNs to sparse datasets. In this paper, we study the behaviour of spiking NNs in handling NLP datasets and the sparsity in their data representation. Then, we propose a novel framework for spiking NN using the concept of stochastic computing. Specifically, instead of generating spike trains with firing rates proportional to the intensity of each value in the feature set separately, the whole feature set is treated as a distribution function and a stochastic spiking train that follow this distribution is generated. This framework reduces the connectivity between NN layers from O(N) to O(logN). Also, it encodes input data differently and make suitable to handle sparse datasets. Finally, the framework achieves high energy efficiency since it uses Integrate and Fire neurons same as conventional spiking NNs. The results show that our proposed stochastic-based SNN achieves nearly the same accuracy as the original DNN on MNIST dataset, and it has better performance than state- of-the-art SNN. Besides that stochastic-based SNN is energy efficient, where the fully connected DNN, the conventional SNN, and the data normalized SNN consume 38.24, 1.83, and 1.85 -times more energy than the stochastic-based SNN, respectively. For sparse datasets, including IMDb and In-House clinical datasets, stochastic-based SNN achieves performance comparable to that of the conventional DNN. However, the conventional spiking NN has a significant decline in classification accuracy.

Alawad, Mohammed↗

DOE OSTI2020

Data optimization for large batch distributed training of deep neural networks

Distributed training in deep learning (DL) is common practice as data and models grow. The current practice for distributed training of deep neural networks faces the challenges of communication bottlenecks when operating at scale, and model accuracy deterioration with an increase in global batch size. Present solutions focus on improving message exchange efficiency as well as implementing techniques to tweak batch sizes and models in the training process. The loss of training accuracy typically happens because the loss function gets trapped in a local minima. We observe that the loss landscape minimization is shaped by both the model and training data and propose a data optimization approach that utilizes machine learning to implicitly smooth out the loss landscape resulting in fewer local minima. Our approach filters out data points which are less important to feature learning, enabling us to speed up the training of models on larger batch sizes to improved accuracy.

Gahlot, Shubhankar↗

DOE OSTI2020

Efficient I/O for Neural Network Training with Compressed Data

FanStore is a shared object store that enables efficient and scalable neural network training on supercomputers. By providing a global cache layer on node-local burst buffers using a compressed representation, it significantly enhances the processing capability of deep learning (DL) applications on existing hardware. In addition, FanStore allows POSIX-compliant file access to the compressed data in user space. We investigate the tradeoff between runtime overhead and data compression ratio using real-world datasets and applications, and propose a compressor selection algorithm to maximize storage capacity given performance constraints. We consider both asynchronous (i.e., with prefetching) and synchronous I/O strategies, and propose mechanisms for selecting compressors for both approaches. Using FanStore, the same storage hardware can host 2-13x more data for example applications without significant runtime overhead. Empirically, our experiments show that FanStore scales to 512 compute nodes with near linear performance scalability.

Zhang, Zhao↗

DOE OSTI2024

Accelerating Lossy and Lossless Compression on Emerging BlueField DPU Architectures

Data compression has become a crucial technique in addressing performance bottlenecks caused by increasing data volumes in High-Performance Computing (HPC), Big Data, and Deep Learning (DL). Despite its potential to boost system performance, recent studies have identified significant challenges with existing compression methods, mainly due to their high computational demands amidst continuously growing data sizes. Concurrently, the advent of Data Processing Units (DPUs), equipped with programmable System-on-Chip (SoC) and specialized compression accelerators, offers a promising opportunity to alter the landscape of data compression. This paper explores the complexities and potential of leveraging NVIDIA BlueField DPUs to accelerate lossy and lossless compression. Towards this, we introduce PEDAL, an innovative library that leverages the hardware capabilities of DPUs to unify and optimize data compression designs. Moreover, we seamlessly co-design PEDAL with the popular MPICH MPI library, demonstrating up to 101x speedup in compression time and 88x decrease in communication latency. Drawing on these achievements, we share our experience with various research communities about accelerating data compression on DPUs in communication-oriented HPC scenarios.

Li, Yuke↗

DOE OSTI2021

Characterizing Machine Learning I/O Workloads on Leadership Scale HPC Systems

High performance computing (HPC) is no longer solely limited to traditional workloads such as simulation and modeling. With the increase in the popularity of machine learning (ML) and deep learning (DL) technologies, we are observing that an increasing number of HPC users are incorporating ML methods into their workflow and scientific discovery processes, across a wide spectrum of science domains such as biology, earth science, and physics. This gives rise to a diverse set of I/O patterns than the traditional checkpoint/restart-based HPC I/O behavior. The details of the I/O characteristics of such ML I/O workloads have not been studied extensively for large-scale leadership HPC systems. This paper aims to fill that gap by providing an in-depth analysis to gain an understanding of the I/O behavior of ML I/O workloads using darshan - an I/O characterization tool designed for lightweight tracing and profiling. We study the darshan logs of more than 23, 000 HPC ML I/O jobs over a time period of one year running on Summit - the second-fastest supercomputer in the world. This paper provides a systematic I/O characterization of ML I/O jobs running on a leadership scale supercomputer to understand how the I/O behavior differs across science domains and the scale of workloads, and analyze the usage of parallel file system and burst buffer by ML I/O workloads.

Paul, Arnab↗

DOE OSTI2021

HYPPO: A Surrogate-Based Multi-Level Parallelism Tool for Hyperparameter Optimization

We present a new software, HYPPO, that enables the automatic tuning of hyperparameters of various deep learning (DL) models. Unlike other hyperparameter optimization (HPO) methods, HYPPO uses adaptive surrogate models and directly accounts for uncertainty in model predictions to find accurate and reliable models that make robust predictions. Using asynchronous nested parallelism, we are able to significantly alleviate the computational burden of training complex architectures and quantifying the uncertainty. HYPPO is implemented in Python and can be used with both TensorFlow and PyTorch libraries. We demonstrate various software features on time-series prediction and image classification problems as well as a scientific application in computed tomography image reconstruction. Finally, we show that (1) we can reduce by an order of magnitude the number of evaluations necessary to find the most optimal region in the hyperparameter space and (2) we can reduce by two orders of magnitude the throughput for such HPO process to complete.

adaptation models↗

DOE OSTI2024

Fault-Tolerant Deep Learning Cache with Hash Ring for Load Balancing in HPC Systems

Large-scale DL on HPC systems like Frontier and Summit uses distributed node-local caching to address scalability and performance challenges. However, as these systems grow more complex, the risk of node failures increases, and current caching approaches lack fault tolerance, jeopardizing large-scale training jobs. We analyzed six months of SLURM job logs from Frontier and found that over 30% of jobs failed after an average of 75 minutes. To address this, we propose fault-tolerance strategies that recache data lost from failed nodes using a hash ring technique for balanced data recaching in the distributed node-local caching, reducing reliance on the PFS. Our extensive evaluations on Frontier showed that the hash ring-based recaching approach reduced training time by approximately 25% compared to the approach that redirects I/O to the PFS after node failures and demonstrated effective load balancing of training data across nodes.

Lee, Seoyeong↗

DOE OSTI2023

Solving Seismic Wave Equations on Variable Velocity Models With Fourier Neural Operator

Here, in the study of subsurface seismic imaging, solving the acoustic wave equation is a pivotal component in existing models. The advancement of deep learning (DL) enables solving partial differential equations (PDEs), including wave equations, by applying neural networks to identify the mapping between the inputs and the solution. This approach can be faster than traditional numerical methods when numerous instances are to be solved. Previous works that concentrate on solving the wave equation by neural networks consider either a single velocity model or multiple simple velocity models, which is restricted in practice. Instead, inspired by the idea of operator learning, this work leverages the Fourier neural operator (FNO) to effectively learn the frequency domain seismic wavefields under the context of variable velocity models. We also propose a new framework paralleled FNO (PFNO) for efficiently training the FNO-based solver given multiple source locations and frequencies. Numerical experiments demonstrate the high accuracy of both FNO and PFNO with complicated velocity models in the OpenFWI datasets. Furthermore, the cross-dataset generalization test verifies that PFNO adapts to out-of-distribution velocity models. Finally, PFNO admits higher computational efficiency on large-scale testing datasets than the traditional finite-difference method. The aforementioned advantages endow the FNO-based solver with the potential to build powerful models for research on seismic waves.

58 GEOSCIENCES↗

DOE OSTI2022

Learning Global Proliferation Expertise Evolution Using AI-Driven Analytics and Public Information

Detecting and anticipating global proliferation expertise and capability evolution from unstructured, noisy, and incomplete public data streams is a highly desired, but extremely challenging task. Here, in this article, we present our pioneering data-driven approach to support the non-proliferation mission to detect and explain the evolution of proliferation expertise and capability development globally from terabytes of publicly available information (PAI), focusing on our knowledge extraction pipeline and descriptive analytics. We first discuss how we fuse nine open-source data streams, including multilingual data, to convert 4 TB of unstructured data to structured knowledge and encode dynamically evolving proliferation expertise representations—content and context graphs. For this, we rely on natural language processing (NLP) and deep learning (DL) models to perform information extraction, topic modeling, and distributed text representation (aka embedding) learning. We then present interactive, usable, and explainable descriptive analytics to refine domain knowledge and present it in a human-understandable form. Finally, we introduce future work avenues that will leverage our dynamic knowledge representations and descriptive analytics to enable predictive and prescriptive inferences to achieve real-time domain understanding and contextual reasoning about global proliferation expertise and capability evolution.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

DOE OSTI2022

Deep learning multiphysics network for imaging CO 2 saturation and estimating uncertainty in geological carbon storage

Multiphysics inversion exploits different types of geophysical data that often complement each other and aims to improve overall imaging resolution and reduce uncertainties in geophysical interpretation. Despite the advantages, traditional multiphysics inversion is challenging because it requires a large amount of computational time and intensive human interactions for preprocessing data and finding trade-off parameters. These issues make it nearly impossible for traditional multiphysics inversion to be applied as a real-time monitoring tool for geological carbon storage. In this paper, we present a deep learning (DL) multiphysics network for imaging CO 2 saturation in real time. The multiphysics network consists of three encoders for analysing seismic, electromagnetic and gravity data and shares one decoder for combining imaging capabilities of the different geophysical data for better predicting CO 2 saturation. The network is trained on pairs of CO 2 label models and multiphysics data so that it can directly image CO 2 saturation. Here we use the bootstrap aggregating method to enhance the imaging accuracy and estimate uncertainties associated with CO 2 saturation images. Using realistic CO 2 label models and multiphysics data derived from the Kimberlina CO 2 storage model, we evaluate the performance of the deep learning multiphysics network and compare its imaging results to those from the deep learning single-physics networks. Our modelling experiments show that the deep learning multiphysics network for seismic, electromagnetic, and gravity data not only improves the imaging accuracy but also reduces uncertainties associated with CO 2 saturation images. Our results also suggest that the deep learning multiphysics network for the non-seismic data (i.e., electromagnetic and gravity) can be used as an effective low-cost monitoring tool in between regular seismic monitoring.

58 GEOSCIENCES↗

DOE OSTI2022

Surrogate Modeling of Nonlinear Dynamic Systems: A Comparative Study

Surrogate models play a vital role in overcoming the computational challenge in designing and analyzing nonlinear dynamic systems, especially in the presence of uncertainty. This paper presents a comparative study of different surrogate modeling techniques for nonlinear dynamic systems. Four surrogate modeling methods, namely, Gaussian process (GP) regression, a long short-term memory (LSTM) network, a convolutional neural network (CNN) with LSTM (CNN-LSTM), and a CNN with bidirectional LSTM (CNN-BLSTM), are studied and compared. All these model types can predict the future behavior of dynamic systems over long periods based on training data from relatively short periods. The multi-dimensional inputs of surrogate models are organized in a nonlinear autoregressive exogenous model (NARX) scheme to enable recursive prediction over long periods, where current predictions replace inputs from the previous time window. Three numerical examples, including one mathematical example and two nonlinear engineering analysis models, are used to compare the performance of the four surrogate modeling techniques. The results show that the GP-NARX surrogate model tends to have more stable performance than the other three deep learning (DL)-based methods for the three particular examples studied. The tuning effort of GP-NARX is also much lower than its deep learning-based counterparts.

42 ENGINEERING↗

DOE OSTI2016

SU-F-I-34: How Does Longitudinal Dose Profile Change with Tube Current Distribution in CT?

Purpose: To investigate how longitudinal dose profile D{sub L}(z) in 30 cm-diameter water cylinder change with tube current (mA) distribution and scan length. Methods: A constant and four variable mA distributions from two previous papers [Dixon et al., Med. Phys. 40, 111920 (14pp.) (2013); Zhang et al., Med. Phys. 41, 091911 (9pp.) (2014)] were adopted in three scan lengths of 10, 28.6, and 50 cm, and all mA distributions had the same average mA over scan ranges. Using the symmetry based dose calculation algorithms and the previously published CT dose equilibration data [Li et al., Med. Phys. 40, 031903 (10pp.) (2013); 41, 111910 (5pp.) (2014)], the authors calculated DL(z) on the phantom central and peripheral axes. Kolmogorov-Smirnov (K-S) test was used to compare the lineshapes of two arbitrary distributions. Results: In constant mA scans, D{sub L}(z) was “bell-shaped”. In variable mA scans, D{sub L}(z) approximately followed the mA lineshape, and the K-S distance generally changed with mA distribution. The distance decreased with scan length, and was larger on the central axis than on the peripheral axis. However, the opposite trends were found in the K-S distance between the D{sub L}(z) distributions of constant and variable mA distributions. Conclusion: Radiation dose from TCM scan is best evaluated using the specific tube current distribution. A constant mA based evaluation may lead to inconsistent longitudinal dose profile with that of TCM scan. Their difference in lineshape is larger on the phantom peripheral axis than on the central axis and increases with scan length. This work confirms that radiation dose in CT depends on not only local mA but also the overall mA distribution and scan length. On the other hand, the concept of regional tube current may be useful when scan length is large, tube current peaks near scan range edge, or the target site is superficial.

60 APPLIED LIFE SCIENCES↗

DOE OSTI2012

Whole-Genome Microarray and Gene Deletion Studies Reveal Regulation of the Polyhydroxyalkanoate Production Cycle by the Stringent Response in Ralstonia eutropha H16

Poly(3-hydroxybutyrate) (PHB) production and mobilization in Ralstonia eutropha are well studied, but in only a few instances has PHB production been explored in relation to other cellular processes. We examined the global gene expression of wild-type R. eutropha throughout the PHB cycle: growth on fructose, PHB production using fructose following ammonium depletion, and PHB utilization in the absence of exogenous carbon after ammonium was resupplied. Our results confirm or lend support to previously reported results regarding the expression of PHB-related genes and enzymes. Additionally, genes for many different cellular processes, such as DNA replication, cell division, and translation, are selectively repressed during PHB production. In contrast, the expression levels of genes under the control of the alternative sigma factor sigma(54) increase sharply during PHB production and are repressed again during PHB utilization. Global gene regulation during PHB production is strongly reminiscent of the gene expression pattern observed during the stringent response in other species. Furthermore, a ppGpp synthase deletion mutant did not show an accumulation of PHB, and the chemical induction of the stringent response with DL-norvaline caused an increased accumulation of PHB in the presence of ammonium. These results indicate that the stringent response is required for PHB accumulation in R. eutropha, helping to elucidate a thus-far-unknown physiological basis for this process.

Brigham, CJ↗

DOE OSTI2018

Photoluminescence Spectra in Polycrystals of Chirally Pure and Racemic Phases of Tryptophan

The photoluminescence spectra of three crystalline phase modifications of aromatic amino acid tryptophan are compared: the left phase of L-tryptophan, the right phase of D-tryptophan, and the racemic phase of DL-tryptophan. The photoluminescence spectra were recorded using a fiber-optic procedure, a small-size FSD-8 spectrometer, and a data processing system that enables comparison of the analyzed spectrum with the spectrum of a reference substance. Photoluminescence was excited by a pulsed-periodic ultraviolet laser at a wavelength of 266 nm.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

DOE OSTI2015

Dust-ion-acoustic double layers in multi-ion dusty plasma

A theoretical investigation has been made on nonplanar (cylindrical and spherical) dust-ionacoustic (DIA) double layers (DLs) in a multi-ion dusty plasma system containing inertial positive and negative ions and arbitrarily charged stationary dust. The dust particles have been considered as arbitrarily (either positively or negatively) charged in order to observe the effects of the dust polarity on the DIA DLs. The ion species were considered to be at different temperatures to observe the effects of the temperatures on that waves. The modified Gardner equation, which has been derived by employing the reductive perturbation method, has been used to analyze time-dependent nonplanar and planar DIA DLs. It has been found that the time evolution of DIA DLs is significantly modified not only by the nonplanar geometry, but also by the polarity, temperature, and mass ratio of the constituent particles. It has been also found that the amplitude of cylindrical DIA DL structures is larger than that of 1D planar ones, but smaller than that of the spherical ones.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

DOE OSTI2022

Models of $J/\varPsi$ photo-production reactions on the nucleon

The J/Ψ photo-production reactions on the nucleon can provide information on the roles of gluons in determining the J/Ψ-nucleon (J/Ψ-N) interactions and the structure of the nucleon. The information on the J/Ψ-N interactions is needed to test lattice QCD (LQCD) calculations and to understand the nucleon resonances such as N*(P c ) recently reported by the LHCb Collaboration. In addition, it is also needed to investigate the production of nuclei with hidden charms and to extract the gluon distributions in nuclei. The main purpose of this article is to review six models of the γ+p→J/Ψ+p reaction which have been and can be applied to analyze the data from Thomas Jefferson National Accelerator Facility (JLab). Here, the formulae for each model are given and used to obtain the results to show the extent to which the available data can be described. The models presented include the Pomeron-exchange model of Donnachie and Landshoff (Pom-DL) and its extensions to include J/Ψ-N potentials extracted from LQCD (Pom-pot) and to also use the constituent quark model (CQM) to account for the quark substructure of J/Ψ (Pom-CQM). The other three models are developed from applying the perturbative QCD approach to calculate the two-gluon exchange using the generalized parton distribution (GPD) of the nucleon (GPD-based), two- and three-gluon exchanges using the parton distribution of the nucleon (2g+3g), and the exchanges of scalar (0 ++ ) and tensor (2 ++ ) glueballs within the holographic formulation (holog). The results of investigating the excitation of the nucleon resonances N*(P c ) in the γ+p→J/Ψ+p reactions are also given. We demonstrate that the differences between these six models can be unambiguously distinguished and the N* can be better studied by using the forthcoming JLab data at large |t| and at energies very near the J/Ψ production threshold. Possible improvements of the considered models are discussed.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

DOE OSTI2015

Optimizing deep learning hyper-parameters through an evolutionary algorithm. In: MLHPC '15 Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, Article No. 4

There has been a recent surge of success in utilizing Deep Learning (DL) in imaging and speech applications for its relatively automatic feature generation and, in particular for convolutional neural networks (CNNs), high accuracy classification abilities. While these models learn their parameters through data-driven methods, model selection (as architecture construction) through hyper-parameter choices remains a tedious and highly intuition driven task. To address this, Multi-node Evolutionary Neural Networks for Deep Learning (MENNDL) is proposed as a method for automating network selection on computational clusters through hyper-parameter optimization performed via genetic algorithms.

Young, Steven R.↗

DOE OSTI2019

STEMDL

The code "STEMDL" provides highly-optimized multi-gpu implementations of DL models of applicability in electron microscopy via the use of mixed-precision arithmetic, native graph construction and evaluation in Tensorflow, and adaptive scaling of learning parameters to avoid numerical instabilities due to reduced precision. STEMDL also uses low-level methods from the Horovod library for gradient communication and reduction and adds an additional optimization by allowing the user to scheduling the gradient reduction operations based on their position within the layers of a neural net. STEMDL provides optimized I/O routines for data staging and data reading/writing in both Tensorflow and Pytorch using multi-processing and asynchronous transfer between host and device. .

Laanait, Nouamane↗