Engineering topics

Landwehr, Joshua B.

Publications and source records attributed to Landwehr, Joshua B..

5 records

DOE OSTI2017

Scaling deep learning workloads: NVIDIA DGX-1/Pascal and Intel Knights Landing

Deep Learning (DL) algorithms have become ubiquitous in data analytics. As a result, major computing vendors --- including NVIDIA, Intel, AMD, and IBM --- have architectural road-maps influenced by DL workloads. Furthermore, several vendors have recently advertised new computing products as accelerating large DL workloads. Unfortunately, it is difficult for data scientists to quantify the potential of these different products. This paper provides a performance and power analysis of important DL workloads on two major parallel architectures: NVIDIA DGX-1 (eight Pascal P100 GPUs interconnected with NVLink) and Intel Knights Landing (KNL) CPUs interconnected with Intel Omni-Path or Cray Aries. Our evaluation consists of a cross section of convolutional neural net workloads: CifarNet, AlexNet, GoogLeNet, and ResNet50 topologies using the Cifar10 and ImageNet datasets. The workloads are vendor-optimized for each architecture. Our analysis indicates that although GPUs provide the highest overall performance, the gap can close for some convolutional networks; and the KNL can be competitive in performance/watt. We find that NVLink facilitates scaling efficiency on GPUs. However, its importance is heavily dependent on neural network architecture. Furthermore, for weak-scaling --- sometimes encouraged by restricted GPU memory --- NVLink is less important.

Gawande, Nitin A.↗

DOE OSTI2017

Designing Scalable Distributed Memory Models: A Case Study

Supercomputers complexity continue to rapidly increase, requiring novel software solutions to aid in addressing the extreme abundance of parallelism, restrictive power constraints, and fault tolerance requirements. While industrial standards like MPI and OpenMP have undergone extensive efforts to prepare for future systems [1, 2], ne grain asynchronous runtimes have demonstrated an innate agility in extracting performance which is otherwise arduous to attain with current execution models

Landwehr, Joshua B.↗

DOE OSTI2016

Extending the Roofline Model for Asynchronous Many-Task Runtimes

New architectural trends and extreme scale parallelism challenge the efficient mapping of applications to large scale and emerging systems. Fine-grained Asynchronous Manytask Runtimes (AMR) offer a unique flexibility to hide the varying latencies of contentious operations, and facilitate the effective exploitation of abundant resources. By decomposing work into smaller asynchronous chunks, the impact of unpredictable operations can be mitigated through a tighter overlapping of tasks. Unfortunately, the cost of this decomposition comes at a potentially steep price in the form of runtime overhead which can at times rival the cost of the computation itself. This leads to the question, how large should a task be? A common practice for application developers is to experimentally determine the granularity of a task after a code has been parallelized. Instead, we propose a new methodology based on an extended Roofline model to provide practical upper bounds on the throughput performance of an application. First, we extend the Roofline model to support not only latency hiding analysis, but also a multidimensional amortized analysis. By combining this new methodology with a serial application and an AMR implementation, we can predict the worst case runtime overhead attribution of individual runtime features prior to the development of parallel code. Thus, this runtimecentric methodology can provide a vehicle for application/runtime codesign by providing a comprehensive bottleneck analysis based on existing runtime features.

Suetterlein, Joshua D.↗

DOE OSTI2016

Asynchronous Runtimes in Action: An Introspective Framework for a Next Gen Runtime

One of the most critical challenges that new high performance systems face is the lack of system software support for these large scale systems. Investment on system stack components is essential in the development, debugging and optimization of the new emerging programming models. These emerging models have the promise to better utilize the vast hardware resources available in current and future systems. To aid in the development of applications and new system stacks, runtimes, as instances of their respective execution models, need to produce facilities to introspect their inner workings and allow an indepth attribution of performance bottlenecks and computational patterns. In other words, the runtime systems need to reduce their opacity to observers so that users of a novel program execution model can adapt their designs to fit the intended model usage, regardless of the layer that they are working on. This design/development loop (akin to co-design) enables synergistic opportunities across the entire computational stack. This paper presents the design and implementation of a simple “gray” box performance attribution harness running inside a fine grain runtime system: the Open Community Runtime (OCR). We showcase what such a framework can indicate regarding the runtime behavior while running at scale. To this end, we have designed a set of synthetic scenarios aimed to test the runtime at their best and worst cases. We present an analysis of the most important runtime features, properties and idiosyncrasies that will affect the development of new runtime features, algorithmic selection, and application development.

high performance distributed runtimes↗

DOE OSTI2016

Application Characterization at Scale: Lessons learned from developing a distributed Open Community Runtime system for High Performance Computing

Since 2012, the U.S. Department of Energy’s X-Stack program has been developing solutions including runtime systems, programming models, languages, compilers, and tools for the Exascale system software to address crucial performance and power requirements. Fine grain programming models and runtime systems show a great potential to efficiently utilize the underlying hardware. Thus, they are essential to many X-Stack efforts. An abundant amount of small tasks can better utilize the vast parallelism available on current and future machines. Moreover, finer tasks can recover faster and adapt better, due to a decrease in state and control. Nevertheless, current applications have been written to exploit old paradigms (such as Communicating Sequential Processor and Bulk Synchronous Parallel processing). To fully utilize the advantages of these new systems, applications need to be adapted to these new paradigms. As part of the applications’ porting process, in-depth characterization studies, focused on both application characteristics and runtime features, need to take place to fully understand the application performance bottlenecks and how to resolve them. This paper presents a characterization study for a novel high performance runtime system, called the Open Community Runtime, using key HPC kernels as its vehicle. This study has the following contributions: one of the first high performance, fine grain, distributed memory runtime system implementing the OCR standard (version 0.99a); and a characterization study of key HPC kernels in terms of runtime primitives running on both intra and inter node environments. Running on a general purpose cluster, we have found up to 1635x relative speed-up for a parallel tiled Cholesky Kernels on 128 nodes with 16 cores each and a 1864x relative speed-up for a parallel tiled Smith-Waterman kernel on 128 nodes with 30 cores.

Application characterization↗