Engineering Papers⌕ Search

Engineering topics

Daily, Jeffrey A.

Publications and source records attributed to Daily, Jeffrey A..

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

Scaling studies for deep learning in Liquid Argon Time Projection Chamber event classification

Measurements in Liquid Argon Time Projection Chamber neutrino detectors feature large, high fidelity event images. Deep learning techniques have been extremely successful in classification tasks of photographs, but their application to these event images is challenging, due to the large size of the events, more two orders of magnitude larger than images found in classical chal- lenges like MNIST or ImageNet. This leads to extremely long training cycles, which slow down the exploration of new network architectures and hyperpa- rameter scans to improve the classification performance. We present studies of scaling an LArTPC classification problem on multiple architectures, spanning multiple nodes. The studies are carried out in simulated events in the MicroBooNE detector.

Strube, Jan F.↗
DOE OSTI2018

Deep Learning for Scientific Discovery

Deep learning has positively impacted fields were perceptive tasks are paramount, such as computer vision and natural language processing. Tremendous strides have also been made in deep reinforcement learning towards robotics and gaming. Given the success in these fields, it is apparent that deep learning may also accelerate scientific discovery. Herein, we discuss several topic areas where modern representation learning is driving innovation. This article presents a) learning non-linear models from data, b) learning scientific laws from models, c) deep learning on graphs, and d) approaches to scalable deep learning.

deep learning, scientific computing, koopman, mate↗
DOE OSTI2017

Transitioning NWChem to the Next Generation of Manycore Machines

The NorthWest Chemistry (NWChem) modeling software is a popular molecular chemistry simulation software that was designed from the start to work on massively parallel processing supercomputers[6, 28, 49]. It contains an umbrella of modules that today includes Self Consistent Field (SCF), second order Mller-Plesset perturbation theory (MP2), Coupled Cluster, multi-conguration selfconsistent eld (MCSCF), selected conguration interaction (CI), tensor contraction engine (TCE) many body methods, density functional theory (DFT), time-dependent density functional theory (TDDFT), real time time-dependent density functional theory, pseudopotential plane-wave density functional theory (PSPW), band structure (BAND), ab initio molecular dynamics, Car-Parrinello molecular dynamics, classical molecular dynamics (MD), QM/MM, AIMD/MM, GIAO NMR, COSMO, COSMO-SMD, and RISM solvation models, free energy simulations, reaction path optimization, parallel in time, among other capabilities[ 22]. Moreover new capabilities continue to be added with each new release.

Environmental Molecular Sciences Laboratory↗
DOE OSTI2017

What does fault tolerant Deep Learning need from MPI?

Deep Learning (DL) algorithms have become the {\em de facto} Machine Learning (ML) algorithm for large scale data analysis. DL algorithms are computationally expensive -- even distributed DL implementations which use MPI require days of training (model learning) time on commonly studied datasets. Long running DL applications become susceptible to faults -- requiring development of a fault tolerant system infrastructure, in addition to fault tolerant DL algorithms. This raises an important question: {\em What is needed from MPI for designing fault tolerant DL implementations?} In this paper, we address this problem for permanent faults. We motivate the need for a fault tolerant MPI specification by an in-depth consideration of recent innovations in DL algorithms and their properties, which drive the need for specific fault tolerance features. We present an in-depth discussion on the suitability of different parallelism types (model, data and hybrid); a need (or lack thereof) for check-pointing of any critical data structures; and most importantly, consideration for several fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI and their applicability to fault tolerant DL implementations. We leverage a distributed memory implementation of Caffe, currently available under the Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches by extending MaTEx-Caffe for using ULFM-based implementation. Our evaluation using the ImageNet dataset and AlexNet neural network topology demonstrates the effectiveness of the proposed fault tolerant DL implementation using OpenMPI based ULFM.

Amatya, Vinay C.↗
DOE OSTI2017

Scaling deep learning workloads: NVIDIA DGX-1/Pascal and Intel Knights Landing

Deep Learning (DL) algorithms have become ubiquitous in data analytics. As a result, major computing vendors --- including NVIDIA, Intel, AMD, and IBM --- have architectural road-maps influenced by DL workloads. Furthermore, several vendors have recently advertised new computing products as accelerating large DL workloads. Unfortunately, it is difficult for data scientists to quantify the potential of these different products. This paper provides a performance and power analysis of important DL workloads on two major parallel architectures: NVIDIA DGX-1 (eight Pascal P100 GPUs interconnected with NVLink) and Intel Knights Landing (KNL) CPUs interconnected with Intel Omni-Path or Cray Aries. Our evaluation consists of a cross section of convolutional neural net workloads: CifarNet, AlexNet, GoogLeNet, and ResNet50 topologies using the Cifar10 and ImageNet datasets. The workloads are vendor-optimized for each architecture. Our analysis indicates that although GPUs provide the highest overall performance, the gap can close for some convolutional networks; and the KNL can be competitive in performance/watt. We find that NVLink facilitates scaling efficiency on GPUs. However, its importance is heavily dependent on neural network architecture. Furthermore, for weak-scaling --- sometimes encouraged by restricted GPU memory --- NVLink is less important.

Gawande, Nitin A.↗
DOE OSTI2017

Evaluating Transactive Controls of Integrated Transmission and Distribution Systems using the Framework for Network Co-Simulation

With an ever-evolving power grid, concerns regarding how to maintain system stability, efficiency, and reliability remain constant because of increasing uncertainties and decreasing rotating inertia. To alleviate some of these concerns, demand response represents a viable solution and is virtually an untapped resource in the current power grid. This work describes a hierarchical control framework that allows coordination between distributed energy resources and demand response. This control framework is composed of two control layers: a coordination layer that ensures aggregations of resources are coordinated to achieve system objectives and a device layer that controls individual resources to assure the predetermined power profile is tracked in real time. Large-scale simulations are executed to study the hierarchical control, requiring advancements in simulation capabilities. Technical advancements necessary to investigate and answer control interaction questions, including the Framework for Network Co-Simulation platform and Arion modeling capability, are detailed. Insights into the interdependencies of controls across a complex system and how they must be tuned, as well as validation of the effectiveness of the proposed control framework, are yielded using a large-scale integrated transmission system model coupled with multiple distribution systems.

demand response↗
DOE OSTI2017

Adaptive Neuron Apoptosis for Accelerating Deep Learning on Large Scale Systems

Machine Learning and Data Mining (MLDM) algorithms are becoming ubiquitous in {\em model learning} from the large volume of data generated using simulations, experiments and handheld devices. Deep Learning algorithms -- a class of MLDM algorithms -- are applied for automatic feature extraction, and learning non-linear models for unsupervised and supervised algorithms. Naturally, several libraries which support large scale Deep Learning -- such as TensorFlow and Caffe -- have become popular. In this paper, we present novel techniques to accelerate the convergence of Deep Learning algorithms by conducting low overhead removal of redundant neurons -- {\em apoptosis} of neurons -- which do not contribute to model learning, during the training phase itself. We provide in-depth theoretical underpinnings of our heuristics (bounding accuracy loss and handling apoptosis of several neuron types), and present the methods to conduct adaptive neuron apoptosis. We implement our proposed heuristics with the recently introduced TensorFlow and using its recently proposed extension with MPI. Our performance evaluation on two difference clusters -- one connected with Intel Haswell multi-core systems, and other with nVIDIA GPUs -- using InfiniBand, indicates the efficacy of the proposed heuristics and implementations. Specifically, we are able to improve the training time for several datasets by 2-3x, while reducing the number of parameters by 30x (4-5x on average) on datasets such as ImageNet classification. For the Higgs Boson dataset, our implementation improves the accuracy (measured by Area Under Curve (AUC)) for classification from 0.88/1 to 0.94/1, while reducing the number of parameters by 3x in comparison to existing literature, while achieving a 2.44x speedup in comparison to the default (no apoptosis) algorithm.

Siegel, Charles M.↗
DOE OSTI2016

On the Impact of Widening Vector Registers on Sequence Alignment

Vector extensions, such as SSE, have been part of the x86 since the 1990s, with applications in graphics, signal processing, and scientific applications. Although many algorithms and applications can naturally benefit from automatic vectorization techniques, there are still many that are difficult to vectorize due to their dependence on irregular data structures, dense branch operations, or data dependencies. Sequence alignment, one of the most widely used operations in bioinformatics workflows, has a computational footprint that features complex data dependencies. In this paper, we demonstrate that the trend of widening vector registers adversely affects the state-of-the-art sequence alignment algorithm based on striped data layouts. We present a practically efficient SIMD implementation of a parallel scan based sequence alignment algorithm that can better exploit wider SIMD units. We conduct comprehensive workload and use case analyses to characterize the relative behavior of the striped and scan approaches and identify the best choice of algorithm based on input length and SIMD width.

Needleman-Wunsch↗
DOE OSTI2016

Parasail: SIMD C library for global, semi-global, and local pairwise sequence alignments

Sequence alignment algorithms are a key component of many bioinformatics applications. Though various fast Smith-Waterman local sequence alignment implementations have been developed for x86 CPUs, most are embedded into larger database search tools. In addition, fast implementations of Needleman-Wunsch global sequence alignment and its semi-global variants are not as widespread. This article presents the first software library for local, global, and semi-global pairwise intra-sequence alignments and improves the performance of previous intra-sequence implementations. As a result, a faster intra-sequence pairwise alignment implementation is described and benchmarked. Using a 375 residue query sequence a speed of 136 billion cell updates per second (GCUPS) was achieved on a dual Intel Xeon E5-2670 12-core processor system, the highest reported for an implementation based on Farrar’s ’striped’ approach. When using only a single thread, parasail was 1.7 times faster than Rognes’s SWIPE. For many score matrices, parasail is faster than BLAST. The software library is designed for 64 bit Linux, OS X, or Windows on processors with SSE2, SSE41, or AVX2. Source code is available from https://github.com/jeffdaily/parasail under the Battelle BSD-style license. In conclusion, applications that require optimal alignment scores could benefit from the improved performance. For the first time, SIMD global, semi-global, and local alignments are available in a stand-alone C library.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2015

Global Arrays

Global Arrays (GA) is a distributed-memory programming model that allows for shared-memory-style programming combined with one-sided communication, to create a set of tools that combine high performance with ease-of-use. GA exposes a relatively straightforward programming abstraction, while supporting fully-distributed data structures, locality of reference, and high-performance communication. GA was originally formulated in the early 1990’s to provide a communication layer for the Northwest Chemistry (NWChem) suite of chemistry modeling codes that was being developed concurrently.

global arrays↗
DOE OSTI2015

Scalable Parallel Methods for Analyzing Metagenomics Data at Extreme Scale

The field of bioinformatics and computational biology is currently experiencing a data revolution. The exciting prospect of making fundamental biological discoveries is fueling the rapid development and deployment of numerous cost-effective, high-throughput next-generation sequencing technologies. The result is that the DNA and protein sequence repositories are being bombarded with new sequence information. Databases are continuing to report a Moore’s law-like growth trajectory in their database sizes, roughly doubling every 18 months. In what seems to be a paradigm-shift, individual projects are now capable of generating billions of raw sequence data that need to be analyzed in the presence of already annotated sequence information. While it is clear that data-driven methods, such as sequencing homology detection, are becoming the mainstay in the field of computational life sciences, the algorithmic advancements essential for implementing complex data analytics at scale have mostly lagged behind. Sequence homology detection is central to a number of bioinformatics applications including genome sequencing and protein family characterization. Given millions of sequences, the goal is to identify all pairs of sequences that are highly similar (or “homologous”) on the basis of alignment criteria. While there are optimal alignment algorithms to compute pairwise homology, their deployment for large-scale is currently not feasible; instead, heuristic methods are used at the expense of quality. In this dissertation, we present the design and evaluation of a parallel implementation for conducting optimal homology detection on distributed memory supercomputers. Our approach uses a combination of techniques from asynchronous load balancing (viz. work stealing, dynamic task counters), data replication, and exact-matching filters to achieve homology detection at scale. Results for a collection of 2.56M sequences show parallel efficiencies of ~75-100% on up to 8K cores, representing a time-to-solution of 33 seconds. We extend this work with a detailed analysis of single-node sequence alignment performance using the latest CPU vector instruction set extensions. Preliminary results reveal that current sequence alignment algorithms are unable to fully utilize widening vector registers.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2015

Diagnosing the Causes and Severity of One-sided Message Contention

Two trends suggest network contention for one-sided messages is poised to become a performance problem that concerns application developers: an increased interest in one-sided programming models and a rising ratio of hardware threads to network injection bandwidth. Unfortunately, it is difficult to reason about network contention and one-sided messages because one-sided tasks can either decrease or increase contention. We present effective and portable techniques for diagnosing the causes and severity of one-sided message contention. To detect that a message is affected by contention, we maintain statistics representing instantaneous (non-local) network resource demand. Using lightweight measurement and modeling, we identify the portion of a message's latency that is due to contention and whether contention occurs at the initiator or target. We attribute these metrics to program statements in their full static and dynamic context. We characterize contention for an important computational chemistry benchmark on InfiniBand, Cray Aries, and IBM Blue Gene/Q interconnects. We pinpoint the sources of contention, estimate their severity, and show that when message delivery time deviates from an ideal model, there are other messages contending for the same network links. With a small change to the benchmark, we reduce contention up to 50% and improve total runtime as much as 20%.

Network Contention, Performance Analysis, Dynamic ↗
DOE OSTI2015

A Global Climate Model Agent for High Spatial and Temporal Resolution Data

Fine cell granularity in modern climate models can produce terabytes of data in each snapshot, causing significant I/O overhead. To address this issue, a method of reducing the I/O latency of high-resolution climate models by identifying and selectively outputting regions of interest is presented. Working with a Global Cloud Resolving Model and running with up to 10240 processors on a Cray XE6, this method provides significant I/O bandwidth reduction depending on the frequency of writes and size of the region of interest. The implementation challenges of determining global parameters in a strictly core-localized model and properly formatting output files that only contain subsections of the global grid are addressed, as well as the overall bandwidth impact and benefits of the method. The gains in I/O throughput provided by this method allow dual output rates for high-resolution climate models: a low-frequency global snapshot as well as a high-frequency regional snapshot when events of particular interest occur.

global cloud resolving model↗
DOE OSTI2014

Co-Simulation of Detailed Whole Building with the Power System to Study Smart Grid Applications

Modernization of the power system in a way that ensures a sustainable energy system is arguably one of the most pressing concerns of our time. Buildings are important components in the power system. First, they are the main consumers of electricity and secondly, they do not have constant energy demand. Conventionally, electricity has been difficult to store and should be consumed as it is generated. Therefore, maintaining the demand and supply is critical in the power system. However, to reduce the complexity of power models, buildings (i.e., end-use loads) are traditionally modeled and represented as aggregated “dumb” nodes in the power system. This means we lack effective detailed whole building energy models that can support requirements and emerging technologies of the smart power grid. To gain greater insight into the relationship between building energy demand and power system performance, it is important to constitute a co-simulation framework to support detailed building energy modeling and simulation within the power system to study capabilities promised by the modern power grid. This paper discusses ongoing work at Pacific Northwest National Laboratory and presents underlying tools and framework needed to enable co-simulation of building, building energy systems and their control in the power system to study applications such as demand response, grid-based HVAC control, and deployment of buildings for ancillary services. The optimal goal is to develop an integrated modeling and simulation platform that is flexible, reusable, and scalable. Results of this work will contribute to future building and power system studies, especially those related to the integrated ‘smart grid’. Results are also expected to advance power resiliency and local (micro) scale grid studies where several building and renewable energy systems transact energy directly. This paper also reviews some applications that can be supported and studied using the framework introduced to understand their implications before they can be successfully implemented in the power system.

Makhmalbaf, Atefe↗
DOE OSTI2014

On the Suitability of MPI as a PGAS Runtime

Partitioned Global Address Space (PGAS) models are emerging as a popular alternative to MPI models for designing scalable applications. At the same time, MPI remains a ubiquitous communication subsystem due to its standardization, high performance, and availability on leading platforms. In this paper, we explore the suitability of using MPI as a scalable PGAS communication subsystem. We focus on the Remote Memory Access (RMA) communication in PGAS models which typically includes {\em get, put,} and {\em atomic memory operations}. We perform an in-depth exploration of design alternatives based on MPI. These alternatives include using a semantically-matching interface such as MPI-RMA, as well as not-so-intuitive interfaces such as MPI two-sided with a combination of multi-threading and dynamic process management. With an in-depth exploration of these alternatives and their shortcomings, we propose a novel design which is facilitated by the data-centric view in PGAS models. This design leverages a combination of highly tuned MPI two-sided semantics and an automatic, user-transparent split of MPI communicators to provide asynchronous progress. We implement the asynchronous progress ranks approach and other approaches within the Communication Runtime for Exascale which is a communication subsystem for Global Arrays. Our performance evaluation spans pure communication benchmarks, graph community detection and sparse matrix-vector multiplication kernels, and a computational chemistry application. The utility of our proposed PR-based approach is demonstrated by a 2.17x speed-up on 1008 processors over the other MPI-based designs.

Computational modeling↗
DOE OSTI2014

A Scientific Data Processing Framework for Time Series NetCDF Data

ARM Data Integrator (ADI) is a framework to streamline the development of scientific algorithms that analyze time-series NetCDF data, and to improve the content and consistency of the output data products produced by these algorithms. ADI achieves these goals by automating the process of retrieving and preparing data for analysis, supporting the definition of output data products through a graphical interface, and providing a modular, flexible software development architecture. The input data, preprocessing, and output data specifications are defined through a graphical interface and stored in a database. ADI also includes a workflow for data integration, a library of software modules to support the workflow, and a source code generator that produces C, IDL and Python templates. Data preparation support includes automated retrieval of data from input files, merging the retrieved data into appropriately sized chunks, and transformation of the data onto a common coordinate system grid. Through the graphical interface, users can view the details of both their data products and those in the ARM catalog. The variable and attribute definitions of the existing data products can be used to build new output data products. In addition, the rules that make up the ARM archive’s data standards are laid on top of the view of the new data product providing the user with a visual cue indicating where their output violates an archive standard. The necessary configurations are stored in a database that is accessed by the ADI libraries. This paper discusses the ADI framework, its supporting components, and how ADI can significantly decrease the time and cost of implementing scientific algorithms while improving the ability of scientists to disseminate their results.

Atmospheric Science↗
DOE OSTI2014

Synchronization Algorithms for Co-Simulation of Power Grid and Communication Networks

The ongoing modernization of power grids consists of integrating them with communication networks in order to achieve robust and resilient control of grid operations. To understand the operation of the new smart grid, one approach is to use simulation software. Unfortunately, current power grid simulators at best utilize inadequate approximations to simulate communication networks, if at all. Cooperative simulation of specialized power grid and communication network simulators promises to more accurately reproduce the interactions of real smart grid deployments. However, co-simulation is a challenging problem. A co-simulation must manage the exchange of informa- tion, including the synchronization of simulator clocks, between all simulators while maintaining adequate computational perfor- mance. This paper describes two new conservative algorithms for reducing the overhead of time synchronization, namely Active Set Conservative and Reactive Conservative. We provide a detailed analysis of their performance characteristics with respect to the current state of the art including both conservative and optimistic synchronization algorithms. In addition, we provide guidelines for selecting the appropriate synchronization algorithm based on the requirements of the co-simulation. The newly proposed algorithms are shown to achieve as much as 14% and 63% im- provement, respectively, over the existing conservative algorithm.

co-simulation↗