Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Message passing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Demonstrating UPC++/Kokkos Interoperability in a Heat Conduction Simulation (Extended Abstract)

We describe the replacement of MPI with UPC++ in an existing Kokkos code that simulates heat conduction within a rectangular 3D object, as well as an analysis of the new code’s performance on CUDA accelerators. The key challenges were packing the halos in Kokkos data structures in a way that allowed for UPC++ remote memory access, and streamlining synchronization costs. Additional UPC++ abstractions used included global pointers, distributed objects, remote procedure calls, and futures. We also make use of the device allocator concept to facilitate data management in memory with unique properties, such as GPUs. Our results demonstrate that despite the algorithm’s good semantic match to message passing abstractions, straightforward modifications to use UPC++ communication deliver vastly improved performance and scalability in the common case. We find the one-sided UPC++ version written in a natural way exhibits good performance, whereas the message-passing version written in a straightforward way exhibits performance anomalies. We argue this represents a productivity benefit for one-sided communication models.

Waters, Daniel↗

Notified access in coarray-based hydrodynamics applications on many-core architectures: Design and performance

With the increasing availability of the Remote Direct Memory Access (RDMA) support in computer networks, the so called Partitioned Global Address Space (PGAS) model has evolved in the last few years. Although there are several cases where a PGAS approach can easily solve difficult message passing situations, like in particle tracking and adaptive mesh refinement applications, the producer-consumer pattern, usually adopted in task-based parallelism, can only be implemented inefficiently because of the separation between data transfer and synchronization (which is usually unified in message passing programming models). Here, in this paper, we provide two contributions: (1) we propose an extension for the Fortran language that provides the concept of Notified Access by associating regular coarray variables with event variables. (2) We demonstrate that the MPI extension proposed by foMPI for Notified Access can be used effectively to implement the same concept in a PGAS run-time library like OpenCoarrays. Moreover, for a hydrodynamics mini-application, we found that Fortran 2018 events perform always better than Fortran 2008 sync statements on many-core processors. We finally show how the proposed Notified Access can improve the performance even more.

97 MATHEMATICS AND COMPUTING↗

Ultrascalable petaflop parallel supercomputer

A massively parallel supercomputer of petaOPS-scale includes node architectures based upon System-On-a-Chip technology, where each processing node comprises a single Application Specific Integrated Circuit (ASIC) having up to four processing elements. The ASIC nodes are interconnected by multiple independent networks that optimally maximize the throughput of packet communications between nodes with minimal latency. The multiple networks may include three high-speed networks for parallel algorithm message passing including a Torus, collective network, and a Global Asynchronous network that provides global barrier and notification functions. These multiple independent networks may be collaboratively or independently utilized according to the needs or phases of an algorithm for optimizing algorithm processing performance. The use of a DMA engine is provided to facilitate message passing among the nodes without the expenditure of processing resources at the node.

97 MATHEMATICS AND COMPUTING↗

Accelerating virtual high-throughput ligand docking: current technology and case study on a petascale supercomputer

In this paper, we give the current state of high-throughput virtual screening. We describe a case study of using a task-parallel Message Passing Interface version of AutoDock4 to run a virtual high-throughput screen of one million compounds on the Jaguar Cray XK6 Supercomputer at Oak Ridge National Laboratory. We include a description of scripts developed to increase the efficiency of the predocking file preparation and postdocking analysis. A detailed tutorial, scripts, and source code for this Message Passing Interface version of AutoDock4 are available online at http://www.bio.utk.edu/baudrylab/autodockmpi.htm.

97 MATHEMATICS AND COMPUTING↗

Complexity in scalable computing.

The rich history of scalable computing research owes much to a rapid rise in computing platform scale in terms of size and speed. As platforms evolve, so must algorithms and the software expressions of those algorithms. Unbridled growth in scale inevitably leads to complexity. This special issue grapples with two facets of this complexity: scalable execution and scalable development. The former results from efficient programming of novel hardware with increasing numbers of processing units (e.g., cores, processors, threads or processes). The latter results from efficient development of robust, flexible software with increasing numbers of programming units (e.g., procedures, classes, components or developers). The progression in the above two parenthetical lists goes from the lowest levels of abstraction (hardware) to the highest (people). This issue's theme encompasses this entire spectrum. The lead author of each article resides in the Scalable Computing Research and Development Department at Sandia National Laboratories in Livermore, CA. Their co-authors hail from other parts of Sandia, other national laboratories and academia. Their research sponsors include several programs within the Department of Energy's Office of Advanced Scientific Computing Research and its National Nuclear Security Administration, along with Sandia's Laboratory Directed Research and Development program and the Office of Naval Research. The breadth of interests of these authors and their customers reflects in the breadth of applications this issue covers. This article demonstrates how to obtain scalable execution on the increasingly dominant high-performance computing platform: a Linux cluster with multicore chips. The authors describe how deep memory hierarchies necessitate reducing communication overhead by using threads to exploit shared register and cache memory. On a matrix-matrix multiplication problem, they achieve up to 96% parallel efficiency with a three-part strategy: intra-node multithreading, non-blocking inter-node message passing, and a dedicated communications thread to facilitate concurrent communications and computations. On a quantum chemistry problem, they spawn multiple computation threads and communication threads on each node and use one-sided communications between nodes to minimize wait times. They reduce software complexity by evolving a multi-threaded factory pattern in C++ from a working, message-passing program in C.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Reducing communication in algebraic multigrid with multi-step node aware communication

Algebraic multigrid (AMG) is often viewed as a scalable [Formula: see text] solver for sparse linear systems. Yet, AMG lacks parallel scalability due to increasingly large costs associated with communication, both in the initial construction of a multigrid hierarchy and in the iterative solve phase. This work introduces a parallel implementation of AMG that reduces the cost of communication, yielding improved parallel scalability. It is common in Message Passing Interface (MPI), particularly in the MPI-everywhere approach, to arrange inter-process communication, so that communication is transported regardless of the location of the send and receive processes. Performance tests show notable differences in the cost of intra- and internode communication, motivating a restructuring of communication. In this case, the communication schedule takes advantage of the less costly intra-node communication, reducing both the number and the size of internode messages. Node-centric communication extends to the range of components in both the setup and solve phase of AMG, yielding an increase in the weak and strong scaling of the entire method.

Computer Science↗

Learning local equivariant representations for large-scale atomistic dynamics

Abstract A simultaneously accurate and computationally efficient parametrization of the potential energy surface of molecules and materials is a long-standing goal in the natural sciences. While atom-centered message passing neural networks (MPNNs) have shown remarkable accuracy, their information propagation has limited the accessible length-scales. Local methods, conversely, scale to large simulations but have suffered from inferior accuracy. This work introduces Allegro, a strictly local equivariant deep neural network interatomic potential architecture that simultaneously exhibits excellent accuracy and scalability. Allegro represents a many-body potential using iterated tensor products of learned equivariant representations without atom-centered message passing. Allegro obtains improvements over state-of-the-art methods on QM9 and revMD17. A single tensor product layer outperforms existing deep MPNNs and transformers on QM9. Furthermore, Allegro displays remarkable generalization to out-of-distribution data. Molecular simulations using Allegro recover structural and kinetic properties of an amorphous electrolyte in excellent agreement with ab-initio simulations. Finally, we demonstrate parallelization with a simulation of 100 million atoms.

74 ATOMIC AND MOLECULAR PHYSICS↗

Graph Neural Network for Object Reconstruction in Liquid Argon Time Projection Chambers

This paper presents a graph neural network (GNN) technique for low-level reconstruction of neutrino interactions in a Liquid Argon Time Projection Chamber (LArTPC). GNNs are still a relatively novel technique, and have shown great promise for similar reconstruction tasks in the LHC. In this paper, a multihead attention message passing network is used to classify the relationship between detector hits by labelling graph edges, determining whether hits were produced by the same underlying particle, and if so, the particle type. The trained model is 84% accurate overall, and performs best on the EM shower and muon track classes. The model’s strengths and weaknesses are discussed, and plans for developing this technique further are summarised.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Toward Abstracting the Communication Intent in Applications to Improve Portability and Productivity

Programming with communication libraries such as the Message Passing Interface (MPI) obscures the high-level intent of the communication in an application and makes static communication analysis difficult to do. Compilers are unaware of communication libraries' specifics, leading to the exclusion of communication patterns from any automated analysis and optimizations. To overcome this, communication patterns can be expressed at higher-levels of abstraction and incrementally added to existing MPI applications. In this paper, we propose the use of directives to clearly express the communication intent of an application in a way that is not specific to a given communication library. Our communication directives allow programmers to express communication among processes in a portable way, giving hints to the compiler on regions of computations that can be overlapped with communication and relaxing communication constraints on the ordering, completion and synchronization of the communication imposed by specific libraries such as MPI. The directives can then be translated by the compiler into message passing calls that efficiently implement the intended pattern and be targeted to multiple communication libraries. Thus far, we have used the directives to express point-to-point communication patterns in C, C++ and Fortran applications, and have translated them to MPI and SHMEM.

Mintz, Tiffany M.↗

Toward Abstracting the Communication Intent in Applications to Improve Portability and Productivity

Programming with communication libraries such as the Message Passing Interface (MPI) obscures the high-level intent of the communication in an application and makes static communication analysis difficult to do. Compilers are unaware of communication libraries specifics, leading to the exclusion of communication patterns from any automated analysis and optimizations. To overcome this, communication patterns can be expressed at higher-levels of abstraction and incrementally added to existing MPI applications. In this paper, we propose the use of directives to clearly express the communication intent of an application in a way that is not specific to a given communication library. Our communication directives allow programmers to express communication among processes in a portable way, giving hints to the compiler on regions of computations that can be overlapped with communication and relaxing communication constraints on the ordering, completion and synchronization of the communication imposed by specific libraries such as MPI. The directives can then be translated by the compiler into message passing calls that efficiently implement the intended pattern and be targeted to multiple communication libraries. Thus far, we have used the directives to express point-to-point communication patterns in C, C++ and Fortran applications, and have translated them to MPI and SHMEM.

Mintz, Tiffany M↗

Distributed Macroscopic Traffic Simulation with Open Traffic Models

This paper presents OTM-MPI, an extension of the Open Traffic Models platform (OTM) for running macroscopic traffic simulations in high-performance computing environments. OTM-MPI represents the first open-source, distributed-memory, macroscopic simulation model developed for modern high performance parallel machines and large networks. Macroscopic simulations are appropriate for studying regional traffic scenarios when aggregate trends are of interest, rather than individual vehicle traces. They are also appropriate for studying the routing behavior of classes of vehicles, such as app-informed vehicles. The network partitioning was performed with METIS. Inter-process communication was done with MPI (message-passing interface). Results are provided for two networks: one realistic network which was obtained from Open Street Maps for Chattanooga, TN, and another larger synthetic grid network. The software recorded a speedups of 198x using 256 cores for Chattanooga, and 475x with 1,024 cores for the synthetic network.

macro-scopic traffic simulation↗

A scalable Euler–Lagrange approach for multiphase flow simulation on spectral elements

Multiphase flow can be difficult to simulate with high accuracy due to the wide range of scales associated with various multiphase phenomena. These scales may range from the size of individual particles to the entire domain of interest. Traditionally, large scale systems can only be simulated using averaging approaches that filter out the locations of individual particles. In this work, the Euler–Lagrange method is used to simulate large-scale dense particle systems in which each individual particle is tracked. In order to accomplish this, the highly scalable spectral element code nek5000 has been extended to handle the multiple levels of multiphase coupling in these systems. These levels include what has been called one-, two-, and four-way coupling. Here, each level has been separated to detail the computational impact of each stage. A binned ghost particle algorithm has also been developed to efficiently handle the challenges of two- and four-way coupling in a parallel processing context. The algorithms and their implementations are then shown to scale to 65,536 Message Passing Interface (MPI) ranks in both the strong and weak limits. After this, validation is performed through simulation of a small-scale fluidized bed. Lastly, a large-scale fluidized bed is simulated with 65,536 MPI ranks and is able to capture the unique physics of the onset of fluidization.

Computer Science↗

A Runtime Environment for Supporting Research in Resilient HPC System Software & Tools

The high-performance computing (HPC) community continues to increase the size and complexity of hardware platforms that support advanced scientific workloads. The runtime environment (RTE) is a crucial layer in the software stack for these large-scale systems. The RTE manages the interface between the operating system and the application running in parallel on the machine. The deployment of applications and tools on large-scale HPC computing systems requires the RTE to manage process creation in a scalable manner, support sparse connectivity, and provide fault tolerance. We have developed a new RTE that provides a basis for building distributed execution environments and developing tools for HPC to aid research in system software and resilience. This paper describes the software architecture of the Scalable runTime Component Infrastructure (STCI), which is intended to provide a complete infrastructure for scalable start-up and management of many processes in large-scale HPC systems. We highlight features of the current implementation, which is provided as a system library that allows developers to easily use and integrate STCI in their tools and/or applications. The motivation for this work has been to support ongoing research activities in fault-tolerance for large-scale systems. We discuss the advantages of the modular framework employed and describe two use cases that demonstrate its capabilities: (i) an alternate runtime for a Message Passing Interface (MPI) stack, and (ii) a distributed control and communication substrate for a fault-injection tool.

Computer Science↗

The LISE package: solvers for static and time-dependent superfluid local density approximation equations in three dimensions

Nuclear implementation of the density functional theory (DFT) is at present the only microscopic framework applicable to the whole nuclear landscape. The extension of DFT to superfluid systems in the spirit of the Kohn-Sham approach, the superfluid local density approximation (SLDA) and its extension to time-dependent situations, time-dependent superfluid local density ap- proximation (TDSLDA), have been extensively used to describe various static and dynamical problems in nuclear physics, neutron star crust, and cold atom systems. In this paper, we present the codes that solve the static and time-dependent SLDA equations in three-dimensional coordinate space without any symmetry restriction. These codes are fully parallelized with the message passing interface (MPI) library and take advantage of graphic processing units (GPU) for accelerating execution. The dynamic codes have checkpoint/restart capabilities and for initial conditions one can use any generalized Slater determinant type of wave function. The code can describe a large number of physical problems: nuclear fission, collisions of heavy ions, the interaction of quantized vor- tices with nuclei in the nuclear star crust, excitation of superfluid fermion systems by time dependent external fields, quantum shock waves, domain wall generation and propagation, the dynamics of the Anderson-Bogoliubov-Higgs mode, dynamics of fragmented condensates, vortex rings dynamics, generation and dynamics of quantized vortices, their crossing and recombinations and the incipient phases of quantum turbulence.

Jin, Shi↗

Performance modeling of microsecond scale biological molecular dynamics simulations on heterogeneous architectures

Performance improvements in biomolecular simulations based on molecular dynamics (MD) codes are widely desired. Unfortunately, the factors, which allowed past performance improvements, particularly the microprocessor clock frequencies, are no longer increasing. Hence, novel software and hardware solutions are being explored for accelerating performance of widely used MD codes. In this paper, we describe our efforts on porting, optimizing and tuning of Large-scale Atomic/Molecular Massively Parallel Simulator, a popular MD framework, on heterogeneous architectures: multi-core processors with graphical processing unit (GPU) accelerators. Our implementation is based on accelerating the most computationally expensive non-bonded interaction terms on the GPUs and overlapping the computation on the CPU and GPUs. This functionality is built on top of message passing interface that allows multi-level parallelism to be extracted even at the workstation level with the multi-core CPUs and allows extension of the implementation on GPU-enabled clusters. We hypothesize that the optimal benefit of heterogeneous architectures for applications will come by utilizing all possible resources (for example, CPU-cores and GPU devices on GPU-enabled clusters). Benchmarks for a range of biomolecular system sizes are provided, and an analysis is performed on four generations of NVIDIA's GPU devices. On GPU-enabled Linux clusters, by overlapping and pipelining computation and communication, we observe up to 10-folds application acceleration in multi-core and multi-GPU environments illustrating significant performance improvements. Detailed analysis of the implementation is presented that allows identification of bottlenecks in algorithm, indicating that code optimization and improvements on GPUs could allow microsecond scale simulation throughput on workstations and inexpensive GPU clusters, putting widely desired biologically relevant simulation time-scales within reach of a large user community. In order to systematically optimize simulation throughput and to enable performance prediction, we have developed a parameterized performance model that will allow developers and users to explore the performance potential of future heterogeneous systems for biological simulations.

Agarwal, Pratul K.↗

A dynamic, unified design for dedicated message matching engines for collective and point-to-point communications

The Message Passing Interface (MPI) libraries use message queues to guarantee correct message ordering between communicating processes. Message queues are in the critical path of MPI communications and thus, the performance of message queue operations can have significant impact on the performance of applications. Collective communications are widely used in MPI applications and they can have considerable impact on generating long message queues. In this paper, we propose a unified message matching mechanism that improves the message queue search time by distinguishing messages coming from point-to-point and collective communications and using a distinct message queue data structure for them. For collective operations, it dynamically profiles the impact of each collective call on message queues during the application runtime and uses this information to adapt the message queue data structure for each collective dynamically. Moreover, we use a partner/non-partner message queue data structure for the messages coming from point-to-point communications. The proposed approach can successfully reduce the queue search time while maintaining scalable memory consumption. The evaluation results show that we can obtain up to 5.5x runtime speedup for applications with long list traversals. Moreover, we can gain up to 15% and 94% queue search time improvement for all elements in applications with short and medium list traversals, respectively.

97 MATHEMATICS AND COMPUTING↗

Development Status of the PEBBLES Code for Pebble Mechanics: Improved Physical Models and Speed-up

PEBBLES is a code for simulating the motion of all the pebbles in a pebble bed reactor. Since pebble bed reactors are packed randomly and not precisely placed, the location of the fuel elements in the reactor is not deterministically known. Instead, when determining operating parameters the motion of the pebbles can be simulated and stochastic locations can be found. The PEBBLES code can output information relevant for other simulations of the pebble bed reactors such as the positions of the pebbles in the reactor, packing fraction change in an earthquake, and velocity profiles created by recirculation. The goal for this level three milestone was to speedup the PEBBLES code through implementation on massively parallel computer. Work on this goal has resulted in speeding up both the single processor version and creation of a new parallel version of PEBBLES. Both the single processor version and the parallel running capability of the PEBBLES code have improved since the fiscal year start. The hybrid MPI/OpenMP PEBBLES version was created this year to run on the increasingly common cluster hardware profile that combines nodes with multiple processors that share memory and a cluster of nodes that are networked together. The OpenMP portions use the Open Multi-Processing shared memory parallel processing model to split the task across processors in a single node that shares memory. The Message Passing Interface (MPI) portion uses messages to communicate between different nodes over a network. The following are wall clock speed up for simulating an NGNP-600 sized reactor. The single processor version runs 1.5 times faster compared to the single processor version at the beginning of the fiscal year. This speedup is primarily due to the improved static friction model described in the report. When running on 64 processors, the new MPI/OpenMP hybrid version has a wall clock speed up of 22 times compared to the current single processor version. When using 88 processors, a speed up of 23 times is achieved. This speedup and other improvements of PEBBLES combine to make PEBBLES more capable and more useful for simulation of a pebble bed reactor. This report details the implementation and effects of the speedup work done over the course of the fiscal year.

10 SYNTHETIC FUELS↗

Development Status of the PEBBLES Code for Pebble Mechanics: Improved Physical Models and Speed-up

PEBBLES is a code for simulating the motion of all the pebbles in a pebble bed reactor. Since pebble bed reactors are packed randomly and not precisely placed, the location of the fuel elements in the reactor is not deterministically known. Instead, when determining operating parameters the motion of the pebbles can be simulated and stochastic locations can be found. The PEBBLES code can output information relevant for other simulations of the pebble bed reactors such as the positions of the pebbles in the reactor, packing fraction change in an earthquake, and velocity profiles created by recirculation. The goal for this level three milestone was to speedup the PEBBLES code through implementation on massively parallel computer. Work on this goal has resulted in speeding up both the single processor version and creation of a new parallel version of PEBBLES. Both the single processor version and the parallel running capability of the PEBBLES code have improved since the fiscal year start. The hybrid MPI/OpenMP PEBBLES version was created this year to run on the increasingly common cluster hardware profile that combines nodes with multiple processors that share memory and a cluster of nodes that are networked together. The OpenMP portions use the Open Multi-Processing shared memory parallel processing model to split the task across processors in a single node that shares memory. The Message Passing Interface (MPI) portion uses messages to communicate between different nodes over a network. The following are wall clock speed up for simulating an NGNP-600 sized reactor. The single processor version runs 1.5 times faster compared to the single processor version at the beginning of the fiscal year. This speedup is primarily due to the improved static friction model described in the report. When running on 64 processors, the new MPI/OpenMP hybrid version has a wall clock speed up of 22 times compared to the current single processor version. When using 88 processors, a speed up of 23 times is achieved. This speedup and other improvements of PEBBLES combine to make PEBBLES more capable and more useful for simulation of a pebble bed reactor. This report details the implementation and effects of the speedup work done over the course of the fiscal year.

10 SYNTHETIC FUELS↗