Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Fault Tolerant Systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

CIFTS : A coordinated infrastructure for fault-tolerant systems.

In the next few years SciDAC applications will utilize petascale systems with tens to hundreds of thousands of processors, hundreds of I/O nodes, and thousands of disks. This leap of two orders of magnitude in scale from today's typical systems is causing a critical gap in fault management of these systems. The fault management issues for these emerging systems are well beyond the scope of today's common infrastructure and practice. Currently, systems software components for large-scale machines remain largely independent in their fault awareness and notification strategies. Faults can arise not just from the hardware but also from the OS, middleware, libraries, and application levels. Petascale applications that are evolving to utilize these platforms face many new challenges. With the CIFTS initiative, we aim to provide a coordinated infrastructure that will enable Fault Tolerant Systems to adapt to faults occuring in the operating environment in a holistic manner. Our approach will be to design a reference implementation of a fault awareness and notification backplane to provide common uniform event handling and notification mechanisms for fault-aware libraries and middleware; create an interface specification that allows libraries, run- time systems, and applications to connect to and use the fault-tolerance backplane; and extend key libraries and applications to validate the interface choices and to form the critical mass necessary for adoption in the community.

99 GENERAL AND MISCELLANEOUS↗

Award ER25750: Coordinated Infrastructure for Fault Tolerance Systems Indiana University Final Report

The main purpose of the Coordinated Infrastructure for Fault Tolerance in Systems initiative has been to conduct research with a goal of providing end-to-end fault tolerance on a systemwide basis for applications and other system software. While fault tolerance has been an integral part of most high-performance computing (HPC) system software developed over the past decade, it has been treated mostly as a collection of isolated stovepipes. Visibility and response to faults has typically been limited to the particular hardware and software subsystems in which they are initially observed. Little fault information is shared across subsystems, allowing little flexibility or control on a system-wide basis, making it practically impossible to provide cohesive end-to-end fault tolerance in support of scientific applications. As an example, consider faults such as communication link failures that can be seen by a network library but are not directly visible to the job scheduler, or consider faults related to node failures that can be detected by system monitoring software but are not inherently visible to the resource manager. If information about such faults could be shared by the network libraries or monitoring software, then other system software, such as a resource manager or job scheduler, could ensure that failed nodes or failed network links were excluded from further job allocations and that further diagnosis could be performed. As a founding member and one of the lead developers of the Open MPI project, our efforts over the course of this project have been focused on making Open MPI more robust to failures by supporting various fault tolerance techniques, and using fault information exchange and coordination between MPI and the HPC system software stack from the application, numeric libraries, and programming language runtime to other common system components such as jobs schedulers, resource managers, and monitoring tools.

97 MATHEMATICS AND COMPUTING↗

Redundancy Does Not Imply Fault Tolerance: Analysis of Distributed Storage Reactions to File-System Faults

In this paper, we analyze how modern distributed storage systems behave in the presence of file-system faults such as data corruption and read and write errors. We characterize eight popular distributed storage systems and uncover numerous problems related to file-system fault tolerance. We find that modern distributed systems do not consistently use redundancy to recover from file-system faults: a single file-system fault can cause catastrophic outcomes such as data loss, corruption, and unavailability. We also find that the above outcomes arise due to fundamental problems in file-system fault handling that are common across many systems. Finally, our results have implications for the design of next-generation fault-tolerant distributed and cloud storage systems.

97 MATHEMATICS AND COMPUTING↗

Blockchain-based fault tolerant grid operations system

To maintain/improve the distribution system fault-tolerance for a grid with high penetration of distributed energy resources requires improvement, Blockchain can add value to improve fault-tolerant grid operations. That can be achieved using blockchain's core features of distributed consensus-based decision-making process and immutability.

Bereta dos Reis, Fernando↗

Blockchain for Fault-Tolerant Grid Operations

Distribution system fault/failure has a direct impact on customers. Distribution systems are not designed with the contingency of one or more elements failing as the transmission system, however, distribution systems are more likely to have faults in comparison to transmission and generation systems. Most of the distribution systems have a radial design with the protection system assumption of unidirectional power flow. The radial design results in the disconnection of customers for any given component failure. The rapid deployment of distributed energy resources conflicts with the assumption of unidirectional power flow. To maintain/improve the distribution system fault tolerance for a grid with high penetration of distributed energy resources requires improvement. Blockchain can add value to improve fault-tolerant grid operations. That can be achieved using blockchain’s core features of distributed consensus-based decision-making process and immutability. In the process of preparing for an event or system restoration, there is a need for reliable data and system situational awareness. The system situational awareness enables accurate planning for possible scenarios and system restoration after an event, an area where blockchain’s distributed architecture can help. The value of blockchain for a fault-tolerant grid operation is elaborated with three use cases: 1) fault location, isolation, and service restoration; 2) topology identification; and 3) data configuration. These use cases demonstrate how blockchain-based architecture can facilitate bridging the vulnerabilities of the existing implementation.

24 POWER TRANSMISSION AND DISTRIBUTION↗

What does fault tolerant Deep Learning need from MPI?

Deep Learning (DL) algorithms have become the {\em de facto} Machine Learning (ML) algorithm for large scale data analysis. DL algorithms are computationally expensive -- even distributed DL implementations which use MPI require days of training (model learning) time on commonly studied datasets. Long running DL applications become susceptible to faults -- requiring development of a fault tolerant system infrastructure, in addition to fault tolerant DL algorithms. This raises an important question: {\em What is needed from MPI for designing fault tolerant DL implementations?} In this paper, we address this problem for permanent faults. We motivate the need for a fault tolerant MPI specification by an in-depth consideration of recent innovations in DL algorithms and their properties, which drive the need for specific fault tolerance features. We present an in-depth discussion on the suitability of different parallelism types (model, data and hybrid); a need (or lack thereof) for check-pointing of any critical data structures; and most importantly, consideration for several fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI and their applicability to fault tolerant DL implementations. We leverage a distributed memory implementation of Caffe, currently available under the Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches by extending MaTEx-Caffe for using ULFM-based implementation. Our evaluation using the ImageNet dataset and AlexNet neural network topology demonstrates the effectiveness of the proposed fault tolerant DL implementation using OpenMPI based ULFM.

Amatya, Vinay C.↗

Evaluation of Simple Causal Message Logging for Large-Scale Fault Tolerant HPC Systems

The era of petascale computing brought machines with hundreds of thousands of processors. The next generation of exascale supercomputers will make available clusters with millions of processors. In those machines, mean time between failures will range from a few minutes to few tens of minutes, making the crash of a processor the common case, instead of a rarity. Parallel applications running on those large machines will need to simultaneously survive crashes and maintain high productivity. To achieve that, fault tolerance techniques will have to go beyond checkpoint/restart, which requires all processors to roll back in case of a failure. Incorporating some form of message logging will provide a framework where only a subset of processors are rolled back after a crash. In this paper, we discuss why a simple causal message logging protocol seems a promising alternative to provide fault tolerance in large supercomputers. As opposed to pessimistic message logging, it has low latency overhead, especially in collective communication operations. Besides, it saves messages when more than one thread is running per processor. Finally, we demonstrate that a simple causal message logging protocol has a faster recovery and a low performance penalty when compared to checkpoint/restart. Running NAS Parallel Benchmarks (CG, MG and BT) on 1024 processors, simple causal message logging has a latency overhead below 5%.

97 MATHEMATICS AND COMPUTING↗

Fault-tolerance for exascale systems.

Periodic, coordinated, checkpointing to disk is the most prevalent fault tolerance method used in modern large-scale, capability class, high-performance computing (HPC) systems. Previous work has shown that as the system grows in size, the inherent synchronization of coordinated checkpoint/restart (CR) limits application scalability; at large node counts the application spends most of its time checkpointing instead of executing useful work. Furthermore, a single component failure forces an application restart from the last correct checkpoint. Suggested alternatives to coordinated CR include uncoordinated CR with message logging, redundant computation, and RAID-inspired, in-memory distributed checkpointing schemes. Each of these alternatives have differing overheads that are dependent on both the scale and communication characteristics of the application. In this work, using the Structural Simulation Toolkit (SST) simulator, we compare the performance characteristics of each of these resilience methods for a number of HPC application patterns on a number of proposed exascale machines. The result of this work provides valuable guidance on the most efficient resilience methods for exascale systems.

99 GENERAL AND MISCELLANEOUS↗

Coordinated Fault-Tolerance for High-Performance Computing Final Project Report

With the Coordinated Infrastructure for Fault Tolerance Systems (CIFTS, as the original project came to be called) project, our aim has been to understand and tackle the following broad research questions, the answers to which will help the HEC community analyze and shape the direction of research in the field of fault tolerance and resiliency on future high-end leadership systems. Will availability of global fault information, obtained by fault information exchange between the different HEC software on a system, allow individual system software to better detect, diagnose, and adaptively respond to faults? If fault-awareness is raised throughout the system through fault information exchange, is it possible to get all system software working together to provide a more comprehensive end-to-end fault management on the system?

97 MATHEMATICS AND COMPUTING↗

Thermoelectric-Driven Sustainable Sensing and Actuation Systems for Fault-Tolerant Nuclear Incidents

The Fukushima Daiichi nuclear incident in March 2011 represented an unprecedented stress test on the safety and backup systems of a nuclear power plant. The lack of reliable information from key components due to station blackout was a serious setback, leaving sensing, actuation, and reporting systems unable to communicate, and safety was compromised. Although there were several independent backup power sources for required safety function on site, ultimately the batteries were drained and the systems stopped working. If, however, key system components were instrumented with self-powered sensing and actuation packages that could report indefinitely on the status of the system, then critical system information could be obtained while providing core actuation and control during off-normal status for as long as needed. This research project focused on the development of such a self-powered sensing and actuation system. The electrical power is derived from intrinsic heat in the reactor components, which is both reliable and plentiful. The key concept was based around using thermoelectric generators that can be integrated directly onto key nuclear components, including pipes, pump housings, heat exchangers, reactor vessels, and shielding structures, as well as secondary-side components. Thermoelectric generators are solid-state devices capable of converting heat directly into electricity. They are commercially available technology. They are compact, have no moving parts, are silent, and have excellent reliability. The key components to the sensor package include a thermoelectric generator (TEG), microcontroller, signal processing, and a wireless radio package, environmental hardening to survive radiation, flooding, vibration, mechanical shock (explosions), corrosion, and excessive temperature. The energy harvested from the intrinsic heat of reactor components can be then made available to power sensors, provide bi-directional communication, recharge batteries for other safety systems, etc. Such an approach is intrinsically fault tolerant: in the event that system temperatures increase, the amount of available energy will increase, which will make more power available for applications. The system can also be used during normal conditions to provide enhanced monitoring of key system components.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Experimental demonstration of continuous quantum error correction

The storage and processing of quantum information are susceptible to external noise, resulting in computational errors. A powerful method to suppress these effects is quantum error correction. Typically, quantum error correction is executed in discrete rounds, using entangling gates and projective measurement on ancillary qubits to complete each round of error correction. Here we use direct parity measurements to implement a continuous quantum bit-flip correction code in a resource-efficient manner, eliminating entangling gates, ancillary qubits, and their associated errors. An FPGA controller actively corrects errors as they are detected, achieving an average bit-flip detection efficiency of up to 91%. Furthermore, the protocol increases the relaxation time of the protected logical qubit by a factor of 2.7 over the relaxation times of the bare comprising qubits. Our results showcase resource-efficient stabilizer measurements in a multi-qubit architecture and demonstrate how continuous error correction codes can address challenges in realizing a fault-tolerant system.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

AutoCheck: Automatically Identifying Variables for Checkpointing by Data Dependency Analysis

Checkpoint/Restart (C/R) has been widely deployed in numerous HPC systems, Clouds, and industrial data centers, which are typically operated by system engineers. Nevertheless, there is no existing approach that helps system engineers without domain expertise and domain scientists without system fault tolerance knowledge identify those critical variables accounted for correct application execution restoration in a failure for C/R. To address this problem, we propose an analytical model and a tool (AutoCheck) that can automatically identify critical variables to checkpoint for C/R. AutoCheck relies on first, analytically tracking and optimizing data dependency between variables and other application execution state, and second, a set of heuristics that identify critical variables for checkpointing from the refined data dependency graph (DDG). AutoCheck allows programmers to pinpoint critical variables to checkpoint quickly within a few minutes. We evaluate AutoCheck on 13 representative HPC benchmarks, demonstrating that AutoCheck can efficiently identify correct critical variables to checkpoint.

HPC↗

Fault isolation and fault-tolerant control for nonlinear stochastic distribution control systems with multiplicative faults

Here, in this paper, a fault isolation, diagnosis and fault tolerant control algorithm is proposed for nonlinear multiple multiplicative faults stochastic distribution control systems employing Takagi–Sugeno fuzzy system. To obtain the detailed fault information, a fault detection algorithm is introduced to discover the fault occurrence time. Then a fault isolation observer is built to produce the residual, and the error system is separated to subsystems affected only by disturbance and multiplicative faults. Moreover, a fault estimation scheme is presented to obtain the fault magnitude information. When faults occur, the system output probability density function will deviate from the desired distribution. So the model predictive control fault tolerant control scheme is needed to minimize the impact of faults as much as possible to make sure that the post fault output probability density function track the desired probability density function. The validity of the designed algorithm is demonstrated through a simulation example, where the fault tolerant control algorithm ensures that the system output probability density function still track the given output probability density function despite the complex case of multiple multiplicative faults occurring simultaneously.

42 ENGINEERING↗

Scalable distributed consensus to support MPI fault tolerance.

As system sizes increase, the amount of time in which an application can run without experiencing a failure decreases. Exascale applications will need to address fault tolerance. In order to support algorithm-based fault tolerance, communication libraries will need to provide fault-tolerance features to the application. One important fault-tolerance operation is distributed consensus. This is used, for example, to collectively decide on a set of failed processes. This paper describes a scalable, distributed consensus algorithm that is used to support new MPI fault-tolerance features proposed by the MPI 3 Forum's fault-tolerance working group. The algorithm was implemented and evaluated on a 4,096-core Blue Gene/P. The implementation was able to perform a full-scale distributed consensus in 305 {mu}s and scaled logarithmically.

99 GENERAL AND MISCELLANEOUS↗

Model Predictive Fault-Tolerant Tracking Control for PDF Control Systems With Packet Losses

In this article, a fault-tolerant tracking control strategy is investigated for nonlinear probability density function (PDF) control systems with the actuator fault, uncertainties, unknown disturbance, and random packet losses. The control input signal dropout and measurement signal dropouts are described as the independent Bernoulli distribution. An adaptive fault diagnosis (FD) observer based on the Lyapunov function is given to simultaneously estimate the fault, disturbance, and state with packet losses. Furthermore, different from the traditional robust fault-tolerant control (FTC), a new active fault-tolerant tracking controller is designed based on the model predictive control framework, which has better adaptive fault-tolerant performance. Finally, the validity of the proposed FTC method has been proved by a simulation study of a papermaking process.

42 ENGINEERING↗

Understanding checkpointing overheads on massive-scale systems : analysis of the IBM Blue Gene/P system.

Providing fault tolerance in high-end petascale systems, consisting of millions of hardware components and complex software stacks, is becoming an increasingly challenging task. Checkpointing continues to be the most prevalent technique for providing fault tolerance in such high-end systems. Considerable research has focussed on optimizing checkpointing; however, in practice, checkpointing still involves a high-cost overhead for users. In this paper, we study the checkpointing overhead seen by various applications running on leadership-class machines like the IBM Blue Gene/P at Argonne National Laboratory. In addition to studying popular applications, we design a methodology to help users understand and intelligently choose an optimal checkpointing frequency to reduce the overall checkpointing overhead incurred. In particular, we study the Grid-Based Projector-Augmented Wave application, the Carr-Parrinello Molecular Dynamics application, the Nek5000 computational fluid dynamics application and the Parallel Ocean Program application-and analyze their memory usage and possible checkpointing trends on 65,536 processors of the Blue Gene/P system.

99 GENERAL AND MISCELLANEOUS↗

Methods and apparatuses for self-generating fault-tolerant keys in spread-spectrum systems

Self-generating fault-tolerant keys for use in spread-spectrum systems are disclosed. At a communication device, beacon signals are received from another communication device and impulse responses are determined from the beacon signals. The impulse responses are circularly shifted to place a largest sample at a predefined position. The impulse responses are converted to a set of frequency responses in a frequency domain. The frequency responses are shuffled with a predetermined shuffle scheme to develop a set of shuffled frequency responses. A set of phase differences is determined as a difference between an angle of the frequency response and an angle of the shuffled frequency response at each element of the corresponding sets. Each phase difference is quantized to develop a set of secret-key quantized phases and a set of spreading codes is developed wherein each spreading code includes a corresponding phase of the set of secret-key quantized phases.

42 ENGINEERING↗

Collaborative operational fault tolerant control for stochastic distribution control system

Based on a class of industry processes, a distributed fault diagnosis and a collaborative operational fault tolerant control algorithms are presented for an irreversible interconnected stochastic distribution control system with boundary conditions. Different from the existed collaborative fault tolerant controllers, the controller does not need to change after fault occurs, the system tracking performance can be maintained by adjusting the setpoint of the whole system. Simulations are also included to show the effectiveness of the proposed algorithm and encouraging results have been obtained.

Stochastic distribution control, Fault diagnosis, ↗