Sultana, Nawrin | Engineering Papers

DOE OSTI2020

Understanding the use of message passing interface in exascale proxy applications

The Exascale Computing Project (ECP) focuses on the development of future exascale-capable applications. Most ECP applications use the message passing interface (MPI) as their parallel programming model with mini-apps serving as proxies. This paper explores the explicit usage of MPI in such ECP proxy applications. We empirically analyze 14 proxy applications from the ECP Proxy Apps Suite. We use the MPI profiling interface (PMPI) to collect MPI usage patterns in ECP proxy apps. Our analysis shows that a small subset of features from MPI is commonly used in the proxies of exascale-capable applications, even when they reference third-party libraries. Overall, this study is intended to provide a better understanding of the use of MPI in current exascale applications. The findings can help focus software investments made for exascale systems in the MPI middleware including optimization, fault-tolerance, tuning, and hardware-offload.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2019

Failure recovery for bulk synchronous applications with MPI stages

When an MPI program experiences a failure, the most common recovery approach is to restart all processes from a previous checkpoint and to re-queue the entire job. A disadvantage of this method is that, although the failure occurred within the main application loop, live processes must start again from the beginning of the program, along with new replacement processes—this incurs unnecessary overhead for live processes. To avoid such overheads and concomitant delays, we introduce the concept of “MPI Stages.” MPI Stages saves internal MPI state in a separate checkpoint in conjunction with application state. Upon failure, both MPI and application state are recovered, respectively, from their last synchronous checkpoints and continue without restarting the overall MPI job. Live processes roll back only a few iterations within the main loop instead of rolling back to the beginning of the program, while a replacement of failed process restarts and reintegrates, thereby achieving faster failure recovery. As a result, this approach integrates well with large-scale, bulk synchronous applications and checkpoint/restart.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2019

MPI Stages : A Serializable MPI Library

MPI library that supports serializing (and check-pointing) its MPI Stages is a communication library that allows nodes and programs in a computer cluster to communicate via the Message-Passing Interface (MPI). The internal data objects of the library are serializable, i.e., the user can convert the MPI state or objects, such as groups or communicators, into byte streams and save them into a checkpoint file. Later users can load the checkpoint file and de-serialize previous objects into the library. Serialization/de-serialization is useful in MPI applications to recover from node failures.

Ruefenacht, Martin↗

DOE OSTI2018

MPI Stages: Checkpointing MPI State for Bulk Synchronous Applications

When an MPI program experiences a failure, the most common recovery approach is to restart all processes from a previous checkpoint and to re-queue the entire job. A disadvantage of this method is that, although the failure occurred within the main application loop, live processes must start again from the beginning of the program, along with new replacement processes---this incurs unnecessary overhead for live processes. To avoid such overheads and concomitant delays, we introduce the concept of "MPI Stages." MPI Stages saves internal MPI state in a separate checkpoint in conjunction with application state. Upon failure, both MPI and application state are recovered, respectively, from their last synchronous checkpoints and continue without restarting the overall MPI job. Live processes roll back only a few iterations within the main loop instead of rolling back to the beginning of the program, while a replacement of failed process restarts and reintegrates, thereby achieving faster failure recovery. This approach integrates well with large-scale, bulk synchronous applications and checkpoint/restart. In this paper, we identify requirements for production MPI implementations to support state checkpointing with MPI Stages, which includes capturing and managing internal MPI state and serializing and deserializing user handles to MPI objects. We evaluate our fault tolerance approach with a proof-of-concept prototype MPI implementation that includes MPI Stages. We demonstrate its functionality and performance using LULESH and microbenchmarks. Our results show that MPI Stages reduces the recovery time by 13× for LULESH in comparison to checkpoint/restart.

Sultana, Nawrin↗