SEARCH · Engineering Papers

Results for “Message Passing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 163 records · Page 9

All sources NASA NTRS DOE OSTI

DOE OSTI2025

MasQiTT

SAND2025-03471O MasQiTT software is a proof-of-concept code that implements the Secure MQTT protocol. It demonstrates a publish/subscribe model of message passing within a network where the publishers can encrypt messages according to a topic and the subscribers can decrypt those messages. The example code shows an example of deployment of an end-to-end system that includes certificate authority and key management in addition to basic implementation of the protocol. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Fisher, Andrew [Sandia National Lab. (SNL-CA), Liv↗

NASA NTRS2018

Utilizing Testing Frameworks for Launch Control Systems Continuous Integration

Command and control software is an integral part of the launch procedure. The most important part of this type of software is its ability to communicate well with the user and relay information in a correctly formatted way such that the user can understand the data. There is a tool that aides the communication between the different parts of the system, and effectively, the user. This instrument is capable of taking several complex values and ensuring that they are correctly sorted into their distinctive message values and distributed properly among the different facets of the system. This tool will easily translate and publish the data inside of messages in the system to something that is readable and understandable. The tool also allows for transmission of the recorded data to the user, effectively ensuring the communication between different components of the system. As well as keeping track of messages and ensuring that the information contained within each of them reaches the correct location, this tool has the ability to keep track of its own statistics and determine how many messages passed in were erroneous and how many were successfully transmitted. It is able to check and see what the total message failure count is when an invalid message is given, as well as the number of different messages and their respective types passed into the tool. This tool is of great value to the new Space Launch System (SLS). As such, the tool must be thoroughly tested with test cases that, although improbable, are possible, where the tool may not function properly. Testing an interface this complex is necessary to ensure mission safety and create unlikely scenarios where the tool would work as intended, and stretch its limits to test that even under the most uncommon conditions it would still continue to function. This software will be an important part of the control system for the newest spacecraft which will fly deeper into space than humans have ever travelled. It will fly beyond the moon, into deep space to Mars and perhaps set the groundwork for a manned mission even further to create more opportunities for interplanetary and even interstellar travel by humans. This mission relies heavily on software and hardware to ensure the safety of the humans that will be on board and therefore must be checked, exhausting each and every different situation, such that there is not a doubt surrounding the well-being of the humans aboard the rocket. That is why testing is such an important part of the mission. It provides evidence that the systems aboard the rocket and on the launch pad are safe.

Unit Testing↗

DOE OSTI2017

Decaf: Decoupled Dataflows for In Situ High-Performance Workflows

Decaf is a dataflow system for the parallel communication of coupled tasks in an HPC workflow. The dataflow can perform arbitrary data transformations ranging from simply forwarding data to complex data redistribution. Decaf does this by allowing the user to allocate resources and execute custom code in the dataflow. All communication through the dataflow is efficient parallel message passing over MPI. The runtime for calling tasks is entirely message-driven; Decaf executes a task when all messages for the task have been received. Such a messagedriven runtime allows cyclic task dependencies in the workflow graph, for example, to enact computational steering based on the result of downstream tasks. Decaf includes a simple Python API for describing the workflow graph. This allows Decaf to stand alone as a complete workflow system, but Decaf can also be used as the dataflow layer by one or more other workflow systems to form a heterogeneous task-based computing environment. In one experiment, we couple a molecular dynamics code with a visualization tool using the FlowVR and Damaris workflow systems and Decaf for the dataflow. In another experiment, we test the coupling of a cosmology code with Voronoi tessellation and density estimation codes using MPI for the simulation, the DIY programming model for the two analysis codes, and Decaf for the dataflow. Such workflows consisting of heterogeneous software infrastructures exist because components are developed separately with different programming models and runtimes, and this is the first time that such heterogeneous coupling of diverse components was demonstrated in situ on HPC systems.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1997

An Application of the "Virtual Spacecraft" Concept in Evaluation of the Mars Pathfinder Lander Low Gain Antenna

The virtual spacecraft concept is embodied in a set of subsystems, either in the form of hardware or computational models, which together represent all, or a portion of, a spacecraft. For example, the telecommunications transponder may be a hardware prototype while the propulsion system may exist only as a simulation. As the various subsystems are realized in hardware, the spacecraft becomes progressively less virtual. This concept is enabled by JPL's Mission System Testbed which is a set of networked workstations running a message passing operating system called "TRAMEL" which stands for Task Remote Asynchronous Message Exchange Layer. Each simulation on the workstations, which may in fact be hardware controlled by the workstation, "publishes" its operating parameters on TRAMEL and other simulations requiring those parameters as input may "subscribe" to them. In this manner, the whole simulation operates as a single virtual system. This paper describes a simulation designed to evaluate a communications link between the earth and the Mars Pathfinder Lander module as it descends under a parachute through the Martian atmosphere toward the planet's surface. This link includes a transmitter and a low gain antenna on the spacecraft and a receiving antenna and receiver on the earth as well as a simulation of the dynamics of the spacecraft. The transmitter, the ground station antenna, the receiver and the dynamics are all simulated computationally while the spacecraft antenna is implemented in hardware on a very simple spacecraft mockup. The dynamics simulation is a record of one output of the ensemble of outputs of a Monte Carlo simulation of the descent. Additionally, the antenna/spacecraft mock-up system was simulated using APATCH, a shooting and bouncing ray code developed by Demaco, Inc. The antenna simulation, the antenna hardware, and the link simulation are all physically located in different facilities at JPL separated by several hundred meters and are linked via the local area network (LAN).

Pogorzelski, R. J.↗

DOE OSTI2014

Scalable Replay with Partial-Order Dependencies for Message-Logging Fault Tolerance

Deterministic replay of a parallel application is commonly used for discovering bugs or to recover from a hard fault with message-logging fault tolerance. For message passing programs, a major source of overhead during forward execution is recording the order in which messages are sent and received. During replay, this ordering must be used to deterministically reproduce the execution. Previous work in replay algorithms often makes minimal assumptions about the programming model and application in order to maintain generality. However, in many cases, only a partial order must be recorded due to determinism intrinsic in the code, ordering constraints imposed by the execution model, and events that are commutative (their relative execution order during replay does not need to be reproduced exactly). In this paper, we present a novel algebraic framework for reasoning about the minimum dependencies required to represent the partial order for different concurrent orderings and interleavings. By exploiting this theory, we improve on an existing scalable message-logging fault tolerance scheme. The improved scheme scales to 131,072 cores on an IBM BlueGene/P with up to 2x lower overhead than one that records a total order.

fault tolerance↗

DOE OSTI2024

Arroyo Stream Processing Toolset (arroyopy) v0.1.0

Processing event or streaming data presents several technological challenges. A variety of technologies are often used by scientific user facilities. ZMQ is used to stream data and messages in a peer-to-peer fashion. Message brokers like Kafka, Redis Pubsub, EPICS PVA and RabbitMQ are often employed to route and pass messages from instruments to processing workflows. Arroyopy provides an API and structure to flexibly integrate with these tools and incorporate arbitrarily complex processing workflows, letting the hooks to the workflow code be independent of the connection code and hence reusable at a variety of instruments.

Chavez Esparza, Tanny Andrea [Lawrence Berkeley Na↗

NASA NTRS1997

Communication Studies of DMP and SMP Machines

Understanding the interplay between machines and problems is key to obtaining high performance on parallel machines. This paper investigates the interplay between programming paradigms and communication capabilities of parallel machines. In particular, we explicate the communication capabilities of the IBM SP-2 distributed-memory multiprocessor and the SGI PowerCHALLENGEarray symmetric multiprocessor. Two benchmark problems of bitonic sorting and Fast Fourier Transform are selected for experiments. Communication-efficient algorithms are developed to exploit the overlapping capabilities of the machines. Programs are written in Message-Passing Interface for portability and identical codes are used for both machines. Various data sizes and message sizes are used to test the machines' communication capabilities. Experimental results indicate that the communication performance of the multiprocessors are consistent with the size of messages. The SP-2 is sensitive to message size but yields a much higher communication overlapping because of the communication co-processor. The PowerCHALLENGEarray is not highly sensitive to message size and yields a low communication overlapping. Bitonic sorting yields lower performance compared to FFT due to a smaller computation-to-communication ratio.

Sohn, Andrew↗

DOE OSTI2012

A uGNI-based Asynchronous Message-driven Runtime System for Cray Supercomputers with Gemini Interconnect

Gemini, the network for the new Cray XE/XK systems, features low latency, high bandwidth and strong scalability. Its hardware support for remote direct memory access enables efficient implementation of the global address space programming languages. Although the user Generic Network Interface (uGNI) provides a low-level interface for Gemini with support to the message-passing programming model (MPI), it remains challenging to port alternative programming models with scalable performance. CHARM++ is an object-oriented message-driven programming model. Its applications have been shown to scale up to the full Jaguar Cray XT machine. In this paper, we present an implementation of this programming model on uGNI for the Cray XE/XK systems. Several techniques are presented to exploit the uGNI capabilites by reducing memory copy and registration overhead, taking advantage of the persistent communication, and improving intra-node communication. Our microbenchmark results demonstrate that the uGNI-based runtime system outperforms the MPI-based implementation by up to 50% in terms of message latency. For communication intensive applications such as N-Queens, this implementation scales up to 15, 360 cores of a Cray XE6 machine and is 70% faster than the MPI-based implementation. In molecular dynamics application NAMD, the performance is also considerably improved by as much as 18%.

Computer Science↗

NASA NTRS1995

A software bus for thread objects

The authors have implemented a software bus for lightweight threads in an object-oriented programming environment that allows for rapid reconfiguration and reuse of thread objects in discrete-event simulation experiments. While previous research in object-oriented, parallel programming environments has focused on direct communication between threads, our lightweight software bus, called the MiniBus, provides a means to isolate threads from their contexts of execution by restricting communications between threads to message-passing via their local ports only. The software bus maintains a topology of connections between these ports. It routes, queues, and delivers messages according to this topology. This approach allows for rapid reconfiguration and reuse of thread objects in other systems without making changes to the specifications or source code. A layered approach that provides the needed transparency to developers is presented. Examples of using the MiniBus are given, and the value of bus architectures in building and conducting simulations of discrete-event systems is discussed.

Callahan, John R.↗

NASA NTRS1988

Embedding triple-modular redundancy into a hypercube architecture

This paper describes an embedding of Triple Modular Redundancy (TMR) into a binary hypercube. The goal is to improve fault tolerance by masking any single-point faults. Each module of an application task is triplicated and executed in parallel on three nodes of a 2-dimensional subcube (Q2) of the hypercube. Each of these nodes also executes a voter process. The remaining node is used for message passing only. All outputs from the triplicated modules are voted on, and the voting results are transmitted to the appropriate destination. Thus, all interunit messages are also triplicated. We propose an embedding of TMR into a hypercube which can be implemented in a manner transparent to the application program. Subcubes are allocated so that the address space for the TMR units is also a hypercube. Hence, the subcube allocation and intermodule communication schemes are defined to be analogous to the schemes used in the nonredundant system. The embedded system is proven to mask all single-point faults.

Kiskis, Daniel L.↗

DOE OSTI2013

SLOAVx: Scalable LOgarithmic AlltoallV Algorithm for Hierarchical Multicore Systems

Scientific applications use collective communication operations in Message Passing Interface (MPI) for global synchronization and data exchanges. Alltoall and AlltoallV are two important collective operations. They are used by MPI jobs to exchange messages among all of MPI processes. AlltoallV is a generalization of Alltoall, supporting messages of varying sizes. However, the existing MPI AlltoallV implementation has linear complexity, i.e., each process has to send messages to all other processes in the job. Such linear complexity can result in sub optimal scalability of MPI applications when they are deployed on millions of cores. To address above challenge, in this paper, we introduce a new Scalable LOgarithmic AlltoallV algorithm, named SLOAV, for MPI AlltoallV collective operation. SLOAV aims to achieve global exchange of small messages of different sizes in a logarithmic number of rounds. Furthermore, given the prevalence of multicore systems with shared memory, we design a hierarchical AlltoallV algorithm based on SLOAV by leveraging the advantages of shared memory, which is referred to as SLOAVx. Compared to SLOAV, SLOAVx significantly reduces the inter-node communication, thus improving the entire system performance and mitigating the impact of message latency. We have implemented and embedded both algorithms in Open MPI. Our evaluation on large-scale computer systems shows that for the 8-byte and 1024-process MPI Alltoallv operation, the SLOAV can reduce the latency by as much as 86.4%, when compared to the state-of-the-art, and SLOAVx can further optimize the SLOAV by up to 83.1% in terms of message latency on multicore systems. In addition, experiments with NAS Parallel Benchmark (NPB) demonstrate that our algorithms are very effective for real-world applications.

Computer Science↗

NASA NTRS1994

Optimal message log reclamation for uncoordinated checkpointing

Uncoordinated checkpointing for message-passing systems allows maximum process autonomy and general nondeterministic execution, but suffers from potential domino effect and the large space overhead for maintaining checkpoints and message logs. Traditionally, it has been assumed that only obsolete checkpoints and message logs before the global recovery line can be garbage-collected. Recently, an approach to identifying all garbage checkpoints based on recovery line transformation and decomposition has been developed. We show in this paper that the same approach can be applied to the problem of identifying all garbage message logs for systems requiring message logging to record in-transit messages. Communication trace-driven simulation for several parallel programs is used to evaluate the proposed algorithm.

Wang, Yi-Min↗

DOE OSTI2011

ConnectX-2 CORE-Direct Enabled Asynchronous Broadcast Collective Communications

This paper describes the design and implementation of InfiniBand (IB) CORE-Direct based blocking and nonblocking broadcast operations within the Cheetah collective operation framework. It describes a novel approach that fully ofFLoads collective operations and employs only user-supplied buffers. For a 64 rank communicator, the latency of CORE-Direct based hierarchical algorithm is better than production-grade Message Passing Interface (MPI) implementations, 150% better than the default Open MPI algorithm and 115% better than the shared memory optimized MVAPICH implementation for a one kilobyte (KB) message, and for eight mega-bytes (MB) it is 48% and 64% better, respectively. Flat-topology broadcast achieves 99.9% overlap in a polling based communication-computation test, and 95.1% overlap for a wait based test, compared with 92.4% and 17.0%, respectively, for a similar Central Processing Unit (CPU) based implementation.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2026

Phloem

Phloem is a Message Passing Interface (MPI) micro-benchmarking suite featuring sub-communicator collectives, methods for finding slow links on MPI interconnects, and point-to-point MPI benchmarks, including a messaging rate benchmark. All of the benchmarks except for ones related exclusively to finding slow links are GPU-aware via the Umpire resource management library.

Moody, AdamT [Lawrence Livermore National Laborato↗

DOE OSTI2011

Improving Multi-Million Virtual Rank MPI Execution in

(MUPI) is a parallel discrete event simulator designed for enabling software-based experimentation via simulated execution across a range of synthetic to unmodified parallel programs using the Message Passing Interface (MPI) with millions of tasks. Here, we report work in progress in improving the efficiency of . Among the issues uncovered are the scaling problems with implementing barriers and intertask message ordering. Preliminary performance shows the possibility of supporting hundreds of virtual MPI ranks per real processor core. Performance improvements of at least 2 are observed, and enable execution of benchmark MPI runs with over 16 million virtual ranks synchronized in a discrete event fashion on as few as 16,128 real cores of a Cray XT5.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2013

Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing

Faults have become the norm rather than the exception for high-end computing on clusters with 10s/100s of thousands of cores. Exacerbating this situation, some of these faults remain undetected, manifesting themselves as silent errors that corrupt memory while applications continue to operate and report incorrect results. This paper studies the potential for redundancy to both detect and correct soft errors in MPI message-passing applications. Our study investigates the challenges inherent to detecting soft errors within MPI application while providing transparent MPI redundancy. By assuming a model wherein corruption in application data manifests itself by producing differing MPI message data between replicas, we study the best suited protocols for detecting and correcting MPI data that is the result of corruption. To experimentally validate our proposed detection and correction protocols, we introduce RedMPI, an MPI library which resides in the MPI profiling layer. RedMPI is capable of both online detection and correction of soft errors that occur in MPI applications without requiring any modifications to the application source by utilizing either double or triple redundancy. Our results indicate that our most efficient consistency protocol can successfully protect applications experiencing even high rates of silent data corruption with runtime overheads between 0% and 30% as compared to unprotected applications without redundancy. Using our fault injector within RedMPI, we observe that even a single soft error can have profound effects on running applications, causing a cascading pattern of corruption in most cases causes that spreads to all other processes. RedMPI's protection has been shown to successfully mitigate the effects of soft errors while allowing applications to complete with correct results even in the face of errors.

Fiala, David J↗

NASA NTRS1993

Scalable Optical-Fiber Communication Networks

Scalable arbitrary fiber extension network (SAFEnet) is conceptual fiber-optic communication network passing digital signals among variety of computers and input/output devices at rates from 200 Mb/s to more than 100 Gb/s. Intended for use with very-high-speed computers and other data-processing and communication systems in which message-passing delays must be kept short. Inherent flexibility makes it possible to match performance of network to computers by optimizing configuration of interconnections. In addition, interconnections made redundant to provide tolerance to faults.

Chow, Edward T.↗

NASA NTRS1989

The Raid distributed database system

Raid, a robust and adaptable distributed database system for transaction processing (TP), is described. Raid is a message-passing system, with server processes on each site to manage concurrent processing, consistent replicated copies during site failures, and atomic distributed commitment. A high-level layered communications package provides a clean location-independent interface between servers. The latest design of the package delivers messages via shared memory in a configuration with several servers linked into a single process. Raid provides the infrastructure to investigate various methods for supporting reliable distributed TP. Measurements on TP and server CPU time are presented, along with data from experiments on communications software, consistent replicated copy control during site failures, and concurrent distributed checkpointing. A software tool for evaluating the implementation of TP algorithms in an operating-system kernel is proposed.

Bhargava, Bharat↗