Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “fault tolerant computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

Redundancy Does Not Imply Fault Tolerance: Analysis of Distributed Storage Reactions to File-System Faults

In this paper, we analyze how modern distributed storage systems behave in the presence of file-system faults such as data corruption and read and write errors. We characterize eight popular distributed storage systems and uncover numerous problems related to file-system fault tolerance. We find that modern distributed systems do not consistently use redundancy to recover from file-system faults: a single file-system fault can cause catastrophic outcomes such as data loss, corruption, and unavailability. We also find that the above outcomes arise due to fundamental problems in file-system fault handling that are common across many systems. Finally, our results have implications for the design of next-generation fault-tolerant distributed and cloud storage systems.

97 MATHEMATICS AND COMPUTING↗

MAX - An advanced parallel computer for space applications

MAX is a fault-tolerant multicomputer hardware and software architecture designed to meet the needs of NASA spacecraft systems. It consists of conventional computing modules (computers) connected via a dual network topology. One network is used to transfer data among the computers and between computers and I/O devices. This network's topology is arbitrary. The second network operates as a broadcast medium for operating system synchronization messages and supports the operating system's Byzantine resilience. A fully distributed operating system supports multitasking in an asynchronous event and data driven environment. A large grain dataflow paradigm is used to coordinate the multitasking and provide easy control of concurrency. It is the basis of the system's fault tolerance and allows both static and dynamical location of tasks. Redundant execution of tasks with software voting of results may be specified for critical tasks. The dataflow paradigm also supports simplified software design, test and maintenance. A unique feature is a method for reliably patching code in an executing dataflow application.

Lewis, Blair F.↗

FOX: A Fault Oblivious Extreme-Scale Execution Environment

The FOX project, funded under the ASCR X-stack I program, developed systems software and runtime libraries for a new approach to the data and work distribution for massively parallel, fault oblivious application execution. Our work was motivated by the premise that exascale computing systems will provide a thousand-fold increase in parallelism and a proportional increase in failure rate relative to today’s machines. To deliver the capability of exascale hardware, the systems software must provide the infrastructure to support existing applications while simultaneously enabling efficient execution of new programming models that naturally express dynamic, adaptive, irregular computation; coupled simulations; and massive data analysis in a highly unreliable hardware environment with billions of threads of execution. Our OS research has prototyped new methods to provide efficient resource sharing, synchronization, and protection in a many-core compute node. We have experimented with alternative task/dataflow programming models and shown scalability in some cases to hundreds of thousands of cores. Much of our software is in active development through open source projects. Concepts from FOX are being pursued in next generation exascale operating systems. Our OS work focused on adaptive, application tailored OS services optimized for multi → many core processors. We developed a new operating system NIX that supports role-based allocation of cores to processes which was released to open source. We contributed to the IBM FusedOS project, which promoted the concept of latency-optimized and throughput-optimized cores. We built a task queue library based on distributed, fault tolerant key-value store and identified scaling issues. A second fault tolerant task parallel library was developed, based on the Linda tuple space model, that used low level interconnect primitives for optimized communication. We designed fault tolerance mechanisms for task parallel computations employing work stealing for load balancing that scaled to the largest existing supercomputers. Finally, we implemented the Elastic Building Blocks runtime, a library to manage object-oriented distributed software components. To support the research, we won two INCITE awards for time on Intrepid (BG/P) and Mira (BG/Q). Much of our work has had impact in the OS and runtime community through the ASCR Exascale OS/R workshop and report, leading to the research agenda of the Exascale OS/R program. Our project was, however, also affected by attrition of multiple PIs. While the PIs continued to participate and offer guidance as time permitted, losing these key individuals was unfortunate both for the project and for the DOE HPC community.

97 MATHEMATICS AND COMPUTING↗

High Performance Spacecraft Computing (HPSC) Middleware Update

High Performance Spacecraft Computing (HPSC) is a joint project between the National Aeronautics and Space Administration (NASA) and Air Force Research Lab (AFRL) to develop a high-performance multi-core radiation hardened flight processor. HPSC offers a new flight computing architecture to meet the needs of NASA missions through 2030 and beyond. Providing on the order of 100X the computational capacity of current flight processors for the same amount of power, the multicore architecture of the HPSC processor, or "Chiplet" provides unprecedented flexibility in a flight computing system by enabling the operating point to be set dynamically, trading among needs for computational performance, energy management and fault tolerance. The HPSC Chiplet is being developed by Boeing under contract to NASA, and is expected to provide prototypes, an evaluation board, system emulators, comprehensive system software, and a software development kit. In addition to the vendor deliverables, the AFRL is funding the development of a flexible Middleware to be developed by NASA Jet Propulsion Laboratory and NASA Goddard Space Flight Center. The HPSC Middleware provides a suite of thirteen high level services to manage the compute, memory and I/O resources of this complex device.This presentation will provide an HPSC project update, an overview of the latest HPSC System Software release, an overview of HPSC Middleware Release 2, and a preview of the third HPSC Middleware release. The presentation will begin with a project update that will provide a look at the high-level changes since the project was introduced at the Flight Software Workshop last year. Next, the presentation will provide an overview of the current suite of HPSC System Software which includes the vendor provided bootloaders, operating systems, emulator, and development tools. Next, the HPSC Middleware progress will be presented, which includes an overview of the features and capabilities of HPSC Middleware Release 2, followed by a look at the reference flight software applications which utilize the Middleware. Finally, the presentation will give a preview of the HPSC Middleware Release 3.

Cudmore, Alan↗

Fault-tolerance for exascale systems.

Periodic, coordinated, checkpointing to disk is the most prevalent fault tolerance method used in modern large-scale, capability class, high-performance computing (HPC) systems. Previous work has shown that as the system grows in size, the inherent synchronization of coordinated checkpoint/restart (CR) limits application scalability; at large node counts the application spends most of its time checkpointing instead of executing useful work. Furthermore, a single component failure forces an application restart from the last correct checkpoint. Suggested alternatives to coordinated CR include uncoordinated CR with message logging, redundant computation, and RAID-inspired, in-memory distributed checkpointing schemes. Each of these alternatives have differing overheads that are dependent on both the scale and communication characteristics of the application. In this work, using the Structural Simulation Toolkit (SST) simulator, we compare the performance characteristics of each of these resilience methods for a number of HPC application patterns on a number of proposed exascale machines. The result of this work provides valuable guidance on the most efficient resilience methods for exascale systems.

99 GENERAL AND MISCELLANEOUS↗

A Redundant Communication Approach to Scalable Fault Tolerance in PGAS Programming Models

Recent trends in high-performance computing point towards increasingly large machines with millions of processing, storage, and networking elements. Unfortunately, the reliability of these machines is inversely proportional to their size, resulting in a system-wide mean-time-between-failures (MTBF) ranging from a few days to a few hours. As such, for long-running applications, the ability to efficiently recover from frequent failures is essential. Traditional forms of fault tolerance, such as checkpoint/restart, suffer from performance issues related to limited I/O and memory bandwidth. In this paper, we present a fault-tolerance mechanism that reduces the cost of failure recovery by maintaining shadow data structures and performing redundant remote memory accesses. We present results from a computational chemistry application running at scale to show that our techniques provide applications with a high degree of fault tolerance and low (2%--4%) overhead for 2048 processors.

99 GENERAL AND MISCELLANEOUS↗

A fault injection experiment using the AIRLAB Diagnostic Emulation Facility

The preparation for, conduct of, and results of a simulation based fault injection experiment conducted using the AIRLAB Diagnostic Emulation facilities is described. An objective of this experiment was to determine the effectiveness of the diagnostic self-test sequences used to uncover latent faults in a logic network providing the key fault tolerance features for a flight control computer. Another objective was to develop methods, tools, and techniques for conducting the experiment. More than 1600 faults were injected into a logic gate level model of the Data Communicator/Interstage (C/I). For each fault injected, diagnostic self-test sequences consisting of over 300 test vectors were supplied to the C/I model as inputs. For each test vector within a test sequence, the outputs from the C/I model were compared to the outputs of a fault free C/I. If the outputs differed, the fault was considered detectable for the given test vector. These results were then analyzed to determine the effectiveness of some test sequences. The results established coverage of selt-test diagnostics, identified areas in the C/I logic where the tests did not locate faults, and suggest fault latency reduction opportunities.

Baker, Robert↗

High-Performance Spaceflight Computing (HPSC) Middleware Overview

High Performance Spacecraft Computing (HPSC) is a joint project between the National Aeronautics and Space Administration (NASA) and Air Force Research Lab (AFRL) to develop a high-performance multi-core radiation hardened flight processor. HPSC offers a new flight computing architecture to meet the needs of NASA missions through 2030 and beyond. Providing on the order of 100X the computational capacity of current flight processors for the same amount of power, the multicore architecture of the HPSC processor, or "Chiplet" provides unprecedented flexibility in a flight computing system by enabling the operating point to be set dynamically, trading among needs for computational performance, energy management and fault tolerance. The HPSC Chiplet is being developed by Boeing under contract to NASA, and is expected to provide prototypes in 2021. The HPSC Chiplet prototypes will be delivered with an evaluation board, system emulators, comprehensive system software, and a software development kit. In addition to the vendor deliverables, the AFRL is funding the development of a flexible Middleware to be developed by NASA Jet Propulsion Laboratory and NASA Goddard Space Flight Center. The HPSC Middleware provides a suite of thirteen high level services to manage the compute, memory and I/O resources of this complex device.This presentation will provide an overview of the HPSC project, including a hardware overview, system software overview, Middleware overview, and mission use cases. The hardware overview will provide a look at the 8 core High Performance Processing Subsystem (HPPS), the Real Time Processing Subsystem (RTPS), the Chiplet Configuration Management Subsystem, on chip peripherals, and high speed I/O. The system software overview will introduce the boot loaders, operating systems, device drivers, and software development environment. The Middleware overview will provide insight into the high-level services that will be provided to help mission developers manage the many resources and configurations made possible with the Chiplet. Finally, the presentation will provide a brief look at the mission use cases that can be enabled with this next generation architecture.

middleware↗

Time Triggered Protocol (TTP) for Integrated Modular Avionics

Traditional avionics computing systems are federated, with each system provided on a number of dedicated hardware units. Federated applications are physically separated from one another and analysis of the systems is undertaken individually. Integrated Modular Avionics (IMA) takes these federated functions and integrates them on a common computing platform in a tightly deterministic distributed real-time network of computing modules in which the different applications can run. IMA supports different levels of criticality in the same computing resource and provides a platform for implementation of fault tolerance through hardware and application redundancy. Modular implementation has distinct benefits in design, testing and system maintainability. This paper covers the requirements for fault tolerant bus systems used to provide reliable communication between IMA computing modules. An overview of the Time Triggered Protocol (TTP) specification and implementation as a reliable solution for IMA systems is presented. Application examples in aircraft avionics and a development system for future space application are covered. The commercially available TTP controller can be also be implemented in an FPGA and the results from implementation studies are covered. Finally future direction for the application of TTP and related development activities are presented.

Motzet, Guenter↗

High-Intensity Radiated Field Fault-Injection Experiment for a Fault-Tolerant Distributed Communication System

Safety-critical distributed flight control systems require robustness in the presence of faults. In general, these systems consist of a number of input/output (I/O) and computation nodes interacting through a fault-tolerant data communication system. The communication system transfers sensor data and control commands and can handle most faults under typical operating conditions. However, the performance of the closed-loop system can be adversely affected as a result of operating in harsh environments. In particular, High-Intensity Radiated Field (HIRF) environments have the potential to cause random fault manifestations in individual avionic components and to generate simultaneous system-wide communication faults that overwhelm existing fault management mechanisms. This paper presents the design of an experiment conducted at the NASA Langley Research Center's HIRF Laboratory to statistically characterize the faults that a HIRF environment can trigger on a single node of a distributed flight control system.

Yates, Amy M.↗

Leveraging Qubit Loss Detection in Fault-Tolerant Quantum Algorithms

Qubit loss errors constitute a dominant source of noise in many quantum hardware systems, particularly in neutral-atom quantum computers. We develop a theoretical framework to effectively detect and correct loss errors in logical algorithms and leverage such loss information in decoding. Considering general quantum error correction codes and logical circuits, we introduce a delayed-erasure decoder for experimentally motivated error models which leverages information from delayed loss detection to accurately correct loss errors, even when the precise moment of the error is unknown. Using this decoder, we identify strategies for detecting and correcting loss errors based on the logical circuit structure. For deep circuits prior to logical measurement, we explore methods to integrate loss detection into syndrome extraction with minimal overhead, identifying optimal strategies depending on the qubit loss fraction in the noise and hardware capabilities. In contrast, we find that many key algorithmic subroutines involve frequent gate teleportation, shortening the circuit depth before logical measurement and naturally replacing qubits with no additional experimental overhead. We simulate this setting using a toy model algorithm for small-angle synthesis and find a significant performance improvement as the loss fraction increases. These results provide a path forward for advancing large-scale fault-tolerant quantum computation in systems with loss error detection.

atoms↗

Design of a modular digital computer system, CDRL no. D001, final design plan

The engineering breadboard implementation for the CDRL no. D001 modular digital computer system developed during design of the logic system was documented. This effort followed the architecture study completed and documented previously, and was intended to verify the concepts of a fault tolerant, automatically reconfigurable, modular version of the computer system conceived during the architecture study. The system has a microprogrammed 32 bit word length, general register architecture and an instruction set consisting of a subset of the IBM System 360 instruction set plus additional fault tolerance firmware. The following areas were covered: breadboard packaging, central control element, central processing element, memory, input/output processor, and maintenance/status panel and electronics.

Easton, R. A.↗

A Byzantine resilient processor with an encoded fault-tolerant shared memory

The memory requirements for ultra-reliable computers are expected to increase due to future increases in mission functionality and operating-system requirements. This increase will have a negative effect on the reliability and cost of the system. Increased memory size will also reduce the ability to reintegrate a channel after a transient fault, since the time required to reintegrate a channel in a conventional fault-tolerant processor is dominated by memory realignment time. A Byzantine Resilient Fault-Tolerant Processor with Fault-Tolerant Shared Memory (FTP/FTSM) is presented as a solution to these problems. The FTSM uses an encoded memory system, which reduces the memory requirement by one-half compared to a conventional quad-FTP design. This increases the reliability and decreases the cost of the system. The realignment problem is also addressed by the FTSM. Because any single error is corrected upon a read from the FTSM, a faulty channel's corrupted memory does not need realignment before reintegration of the faulty channel. A combination of correct-on-access and background scrubbing is proposed to prevent the accumulation of transient errors in the memory. With a hardware-implemented scrubber, the scrubbing cycle time, and therefore the memory fault latency, can be upper-bounded at a small value. This technique increases the reliability of the memory system and facilitates validation of its reliability model.

Butler, Bryan↗

SIFT - A preliminary evaluation

This paper presents the results of a performance evaluation of the SIFT computer system conducted in the NASA AIRLAB facility. The essential system functions are described and compared to both earlier design proposals and subsequent design improvements. The functions supporting fault tolerance are found to consume significant computing resources. With SIFT's specimen task load, scheduled at a 30-Hz rate, the executive tasks such as reconfiguration, clock synchronization and interactive consistency, require 55 percent of the available task slots. Other system overhead (e.g., voting and scheduling) use an average of 50 percent of each remaining task slot.

Palumbo, D. L.↗

Interface Circuits for Self-Checking Microprocessors

Fault-tolerant-microcomputer concept based on enhancing "simple" computer with redundancy and self-checking logic circuits detect hardware faults. Interface and checking logic and redundant processors confer on 16-bit microcomputer ability to check itself for hardware faults. Checking circuitry also checks itself. Concept of self-checking complementary pairs (SCCP's) employed throughout ICL unit.

Rennels, D. A.↗

Phased models for evaluating the performability of computing systems

A phase-by-phase modelling technique is introduced to evaluate a fault tolerant system's ability to execute different sets of computational tasks during different phases of the control process. Intraphase processes are allowed to differ from phase to phase. The probabilities of interphase state transitions are specified by interphase transition matrices. Based on constraints imposed on the intraphase and interphase transition probabilities, various iterative solution methods are developed for calculating system performability.

Wu, L. T.↗

Integrating Fault-Tolerance into the Monte Carlo Application Toolkit

High-performance computer systems have passed the petascale mark, and the next decade will witness a drive towards exascale systems. Extrapolating from current hardware trends it is evident that exascale class systems will require a significantly larger number of cores than current petascale systems, and advanced architectures which include heterogeneous processors are gaining popularity. Such an increase in component count will decrease system mean time to interrupt (MTTI), resulting in greater emphasis on application fault tolerance. The additional memory requirements of exascale systems combined with the slow increase in hard drive write bandwidth will exclude current checkpoint/restart (C/R) techniques as a reliable means of fault tolerance. The current C/R approach to handling failures typically aborts the entire application and requires restarting the application, which is clearly an inefficient solution when utilizing 1000+ processors. Unless improvements are made in resilience and fault tolerance, it is predicted that the work throughput on the largest supercomputers could drop below 10% by 2015. As part of the ”Application Enablement for Next Generation Platforms” milestone, we addressed the shortcomings and limitations of C/R by developing a fault-tolerant component which would be integrated into the Monte Carlo Application Toolkit (MCATK) and demonstrated the ability to continue static K-effective calculations even in the presence of multiple processor failures. In section 2, we provide details about the milestone tasks. Section 3 describes the fault-tolerant framework, its integration into MCATK and its failure recovery scheme. We also describe FT-MPI, a fault-tolerant MPI implementation which provides the capabilities needed for an MPI-based application to survive and continue past failures. Section 4 reports future work that we hope to build upon the current accomplishments.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Concatenated dual displacement code for continuous-variable quantum error correction

The continuous-variable (CV) Gaussian no-go theorem fundamentally limits the suppression of Gaussian displacement errors using only Gaussian gates and states. Prior studies have employed Gottesman-Kitaev-Preskill (GKP) states as ancillary qumodes to suppress small Gaussian displacement errors. However, when the displacement magnitude becomes large, inevitable lattice-crossing errors arise beyond the correctable range of the GKP state. To address this issue, we concatenate the Gaussian-noise-suppression circuit with an outer analog Steane code that corrects such occasional lattice-crossing events as well as other abrupt displacement errors. Contrary to conventional concatenation, which primarily aims to reduce logical error rates, the Steane-GKP duality in encoding provides complementary protection against displacement errors at different scales: The inner GKP layer employs non-Gaussian resources to suppress continuous Gaussian noise and reduce residual variance, while the outer analog Steane code corrects discrete lattice-crossing events that exceed the GKP correctable range. It is precisely this separation of error-mitigation roles that enables CV error correction. In contrast to prior work on concatenating GKP and repetition codes to establish error correction for discrete qubit/qudit encoding, we provide correction in the continuous encoding space. Analytical studies show that, under infinite squeezing, the concatenated code suppresses the variance of Gaussian displacement errors acting on all qumodes by up to 50%, while enabling unbiased correction of lattice-crossing errors with a success probability determined by the ratio between the residual Gaussian error standard deviation and the lattice-crossing magnitude. Even with finite squeezing, the proposed architecture still provides Gaussian-error suppression and lattice-crossing correction. Moreover, the presence of the outer analog Steane code relaxes the squeezing requirement of the inner GKP states, indicating near-term experimental feasibility. This work establishes a viable route toward fault-tolerant continuous-variable quantum computation and provides insight into the design of concatenated CV error-correcting architectures.

quantum error correction↗