Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “fault tolerant computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 757 records · Page 42

Fault Survivability of Lightweight Operating Systems for exascale

Concern is growing in the High-Performance Computing community regarding the reliability of proposed exascale systems. Current research has shown that the expected reliability of these machines will greatly reduce their scalability. In constrast to current fault tolerance methods whose reliability focus is only the application, this project investigates the benefits integrating reliability mechcanisms in the operating system and runtime, as well as the appli- cation. More specifically, this project has three broad contributions in the field: First, using failure logs from current leadership-class high-performance computing systems, we outline the failures common on these large-scale systems. Second, we describe a novel memory pro- tection mechcanism capable of protecting common observed failures that uses the similarity inherrant in many OS and applications state, thereby reducing overheads. Finally, using an analogy with OS jitter, we develop a highly effecient simulator capable predicting the performance of resilience methods at the scales expected for future extreme-scale systems.

97 MATHEMATICS AND COMPUTING↗

Evolution of safety-critical requirements post-launch

This paper reports the results of a small study of requirements changes to the onboard software of three spacecraft subsequent to launch. Only those requirement changes that resulted from post-launch anoma-lies (i.e., during operations) were of interest here, since the goal was to better understand the relation-ship between critical anomalies during operations and how safety-critical requirements evolve. The results of the study were surprising in that anomaly-driven, post-launch requirements changes were rarely due to previous requirements having been incorrect. Instead, changes involved new requirements (1) for the software to handle rare events or (2) for the software to compensate for hardware failures or limitations. The prevalence of new requirements as a result of post-launch anomalies suggests a need for increased requirements-engineering support of maintenance activities in these systems. The results also confirm both the difficulty and the benefits of pursuing requirements completeness, especially in terms of fault tolerance, during development of critical systems.

Software requirements↗

Memory coherence of a sympathetically cooled trapped-ion qubit

We demonstrate sympathetic cooling of a {sup 43}Ca{sup +} trapped-ion 'memory' qubit by a {sup 40}Ca{sup +}'coolant' ion sufficiently near the ground state of motion for fault-tolerant quantum logic, while maintaining coherence of the qubit. This is an essential ingredient in trapped-ion quantum computers. The isotope shifts are sufficient to suppress decoherence and phase shifts of the memory qubit due to the cooling light which illuminates both ions. We measure the qubit coherence during ten cycles of sideband cooling, finding a coherence loss of 3.3% per cooling cycle. The natural limit of the method is O(10{sup -4}) infidelity per cooling cycle.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

An Integrated Risk Assessment Process for Digital Instrumentation and Control Upgrades of Nuclear Power Plants

Most of the existing nuclear power plants (NPPs) in the world rely on traditional analog instrumentation and control (I&C) systems for monitoring, control, and protection functions. With the industrial base largely moving to digital systems, the operation and maintenance of plants involves managing issues including lack of needed analog spare parts, increasing maintenance costs, and the loss of vendor support. Compared with existing analog I&C systems, digital I&C systems have significant functional advantages, such as reliable system performance in terms of accuracy and computational capability, high data handling and storage capabilities to fully measure and display operating conditions, and improved capabilities (e.g., fault tolerance, self-testing, signal validation, process system diagnostics). Therefore, the U.S. nuclear power industry has initiated the replacement of existing, aging analog systems with digital I&C technology, and is developing new designs for advanced plants using digital systems in integrated control rooms to provide modern control and protection systems. However, the qualification of digital I&C systems remains a challenge, especially the issue of software common cause failure (CCF), which has been difficult to address. A CCF is the malfunction of two or more plant components or functions due to a single failure source. CCFs have the potential to generate unanalyzed events or sequences that may not be bounded by previous plant accident analyses, therefore, to challenge the plant safety. Existing analyses on CCF in I&C systems are mainly focusing on hardware failures. With the application and upgrades of new digital I&C systems, software CCFs due to design flaws have become a potential threat to plant safety considering most redundancy designs are using the similar digital platforms or software in the operating and application systems. With complex multi-layer redundancy designs to meet the single failure criterion, these I&C safety systems are of a particular concern in the U.S. Nuclear Regulatory Commission (NRC) licensing procedures. Therefore, there is a need to develop an integrated risk assessment strategy with digital CCF and plant transient responses considered to assure the long-term safety and reliability of vital digital systems and reduce uncertainties in costs, time, and support integration of digital systems in the plant. The overall goal of this project is to deliver a strong technical basis to support effective, licensable, and secure digital I&C technologies for the digital upgrades to existing NPPs. To deal with the expensive licensing justifications from regulatory insights, this technical basis is instructive for nuclear vendors and utilities to effectively lower the costs associated with digital compliance and speed-up industry advances by: (1) defining an integrated risk-informed analysis process for digital I&C upgrades including hazard analysis, reliability analysis, and consequence analysis; (2) applying systematic and risk-informed tools to address CCFs and quantify responding failure probabilities for digital I&C technologies; (3) evaluating the impact of digital failures at the individual level, system level, and plant level; and (4) providing insights and suggestions on designs to manage the risks, thus to support the development, licensing, and deployment of advanced digital I&C technologies on NPPs. Upgrading digital I&C (safety and non-safety related) systems in existing NPPs within a cost-effective and regulatory acceptable way offers the foremost means of performance improvements and cost-reductions for existing NPPs. One key outcome of this project is to perform plant-specific risk assessment to provide a sustainable scientific support for enabling industry to balance the digital-related risks, costs, reliability, and safety.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Computer-aided reliability estimation

Computer-aided reliability estimation (CARE) programs are developed to improve the tools available for estimating the reliability of fault-tolerant systems. A description is presented of a program, called CARE II, which was developed after the first program reported by Mathur (1971). Attention is given to the CARE II reliability model, the CARE II coverage model, and CARE II limitations which are to be rectified in CARE III. It is pointed out that the present coverage model in CARE II is extremely versatile. The major limitation is related to the burden placed on the user to determine the basic parameters from which the coverage calculations are made.

Stiffler, J. J.↗

Efficient Checkpointing of Virtual Machines using Virtual Machine Introspection

Cloud Computing environments rely heavily on system-level virtualization. This is due to the inherent benefits of virtualization including fault tolerance through checkpoint/restart (C/R) mechanisms. Because clouds are the abstraction of large data centers and large data centers have a higher potential for failure, it is imperative that a C/R mechanism for such an environment provide minimal latency as well as a small checkpoint file size. Recently, there has been much research into C/R with respect to virtual machines (VM) providing excellent solutions to reduce either checkpoint latency or checkpoint file size. However, these approaches do not provide both. This paper presents a method of checkpointing VMs by utilizing virtual machine introspection (VMI). Through the usage of VMI, we are able to determine which pages of memory within the guest are used or free and are better able to reduce the amount of pages written to disk during a checkpoint. We have validated this work by using various benchmarks to measure the latency along with the checkpoint size. With respect to checkpoint file size, our approach results in file sizes within 24% or less of the actual used memory within the guest. Additionally, the checkpoint latency of our approach is up to 52% faster than KVM s default method.

Aderholdt, Ferrol↗

Mini-Ckpts: Surviving OS Failures in Persistent Memory

Concern is growing in the high-performance computing (HPC) community on the reliability of future extreme-scale systems. Current efforts have focused on application fault-tolerance rather than the operating system (OS), despite the fact that recent studies have suggested that failures in OS memory are more likely. The OS is critical to a system's correct and efficient operation of the node and processes it governs -- and in HPC also for any other nodes a parallelized application runs on and communicates with: Any single node failure generally forces all processes of this application to terminate due to tight communication in HPC. Therefore, the OS itself must be capable of tolerating failures. In this work, we introduce mini-ckpts, a framework which enables application survival despite the occurrence of a fatal OS failure or crash. Mini-ckpts achieves this tolerance by ensuring that the critical data describing a process is preserved in persistent memory prior to the failure. Following the failure, the OS is rejuvenated via a warm reboot and the application continues execution effectively making the failure and restart transparent. The mini-ckpts rejuvenation and recovery process is measured to take between three to six seconds and has a failure-free overhead of between 3-5% for a number of key HPC workloads. In contrast to current fault-tolerance methods, this work ensures that the operating and runtime system can continue in the presence of faults. This is a much finer-grained and dynamic method of fault-tolerance than the current, coarse-grained, application-centric methods. Handling faults at this level has the potential to greatly reduce overheads and enables mitigation of additional fault scenarios.

Fiala, David↗

HiRel: Hybrid Automated Reliability Predictor (HARP) integrated reliability tool system, (version 7.0). Volume 3: HARP Graphics Oriented (GO) input user's guide

The Hybrid Automated Reliability Predictor (HARP) integrated Reliability (HiRel) tool system for reliability/availability prediction offers a toolbox of integrated reliability/availability programs that can be used to customize the user's application in a workstation or nonworkstation environment. HiRel consists of interactive graphical input/output programs and four reliability/availability modeling engines that provide analytical and simulative solutions to a wide host of highly reliable fault-tolerant system architectures and is also applicable to electronic systems in general. The tool system was designed at the outset to be compatible with most computing platforms and operating systems, and some programs have been beta tested within the aerospace community for over 8 years. This document is a user's guide for the HiRel graphical preprocessor Graphics Oriented (GO) program. GO is a graphical user interface for the HARP engine that enables the drawing of reliability/availability models on a monitor. A mouse is used to select fault tree gates or Markov graphical symbols from a menu for drawing.

Bavuso, Salvatore J.↗

New-Sum: A Novel Online ABFT Scheme For General Iterative Methods

Emerging high-performance computing platforms, with large component counts and lower power margins, are anticipated to be more susceptible to soft errors in both logic circuits and memory subsystems. We present an online algorithm-based fault tolerance (ABFT) approach to efficiently detect and recover soft errors for general iterative methods. We design a novel checksum-based encoding scheme for matrix-vector multiplication that is resilient to both arithmetic and memory errors. Our design decouples the checksum updating process from the actual computation, and allows adaptive checksum overhead control. Building on this new encoding mechanism, we propose two online ABFT designs that can effectively recover from errors when combined with a checkpoint/rollback scheme.

Instantaneous Footprint↗

Reactive system verification case study: Fault-tolerant transputer communication

A reactive program is one which engages in an ongoing interaction with its environment. A system which is controlled by an embedded reactive program is called a reactive system. Examples of reactive systems are aircraft flight management systems, bank automatic teller machine (ATM) networks, airline reservation systems, and computer operating systems. Reactive systems are often naturally modeled (for logical design purposes) as a composition of autonomous processes which progress concurrently and which communicate to share information and/or to coordinate activities. Formal (i.e., mathematical) frameworks for system verification are tools used to increase the users' confidence that a system design satisfies its specification. A framework for reactive system verification includes formal languages for system modeling and for behavior specification and decision procedures and/or proof-systems for verifying that the system model satisfies the system specifications. Using the Ostroff framework for reactive system verification, an approach to achieving fault-tolerant communication between transputers was shown to be effective. The key components of the design, the decoupler processes, may be viewed as discrete-event-controllers introduced to constrain system behavior such that system specifications are satisfied. The Ostroff framework was also effective. The expressiveness of the modeling language permitted construction of a faithful model of the transputer network. The relevant specifications were readily expressed in the specification language. The set of decision procedures provided was adequate to verify the specifications of interest. The need for improved support for system behavior visualization is emphasized.

Crane, D. Francis↗

File I/O for MPI Applications in Redundant Execution Scenarios

As multi-petascale and exa-scale high-performance computing (HPC) systems inevitably have to deal with a number of resilience challenges, such as a significant growth in component count and smaller circuit sizes with lower circuit voltages, redundancy may offer an acceptable level of resilience that traditional fault tolerance techniques, such as checkpoint/restart, do not. Although redundancy in HPC is quite controversial due to the associated cost for redundant components, the constantly increasing number of cores-per-processor is tilting this cost calculation toward a system design where computation, such as for redundancy, is much cheaper and communication, needed for checkpoint/restart, is much more expensive. Recent research and development activities in redundancy for Message Passing Interface (MPI) applications focused on availability/reliability models and replication algorithms. This paper takes a first step toward solving an open research problem associated with running a parallel application redundantly, which is file I/O under redundancy. The approach intercepts file I/O calls made by a redundant application to employ coordination protocols that execute file I/O operations in a redundancy-oblivious fashion when accessing a node-local file system, or in a redundancy-aware fashion when accessing a shared networked file system. A proof-of concept prototype is presented and a number of coordination protocols are described and evaluated. The results show the performance impact for redundantly accessing a shared networked file system, but also demonstrate the capability to regain performance by utilizing MPI communication between replicas and parallel file I/O.

99 GENERAL AND MISCELLANEOUS↗

Peer-to-Peer Communication Trade-Offs for Smart Grid Applications: Preprint

Peer-to-peer energy management systems for smart grids require developers to consider the trade-offs between the amount of communication traffic generated and the quality and speed of convergence of the control algorithms that are deployed. Employing a fully connected communication causes messages to scale exponentially with the number of nodes, while using a sparse connectivity causes less information dissemination leading to degradation of the algorithm performance. The best communication topology for a particular application lies somewhere in between and often requires empirical evaluation by application designers. Existing methods do not put focus on the needs for smart grid applications, which is information dissemination throughout the network and they do not provide a flexible solution for application developers to prototype and deploy different topologies without modifying the application code. This paper introduces a configurable virtual communication topology framework TopLinkMgr, allowing users to specify any chosen communication topology and deploy peer-to-peer applications using it. It also introduces a self-adaptive, fault-tolerant topology management algorithm, Bounded Path Dissemination that can ensure the dissemination of information to all peers within a specified threshold for a sparsely connected topology. Experiments show that the algorithm improves on convergence speed and accuracy over state-of-the-art methods and is also robust against node failures. The results indicate the possibility of achieving a close-to optimal convergence without overloading the network allowing the realization of peer-to-peer control platforms covering larger and more complex power systems.

Bounded Path Dissemination↗

Adapting Secure MultiParty Computation to Support Machine Learning in Radio Frequency Sensor Networks

In this project we developed and validated algorithms for privacy-preserving linear regression using a new variant of Secure Multiparty Computation (MPC) we call "Hybrid MPC" (hMPC). Our variant is intended to support low-power, unreliable networks of sensors with low-communication, fault-tolerant algorithms. In hMPC we do not share training data, even via secret sharing. Thus, agents are responsible for protecting their own local data. Only the machine learning (ML) model is protected with information-theoretic security guarantees against honest-but-curious agents. There are three primary advantages to this approach: (1) after setup, hMPC supports a communication-efficient matrix multiplication primitive, (2) organizations prevented by policy or technology from sharing any of their data can participate as agents in hMPC, and (3) large numbers of low-power agents can participate in hMPC. We have also created an open-source software library named "Cicada" to support hMPC applications with fault-tolerance. The fault-tolerance is important in our applications because the agents are vulnerable to failure or capture. We have demonstrated this capability at Sandia's Autonomy New Mexico laboratory through a simple machine-learning exercise with Raspberry Pi devices capturing and classifying images while flying on four drones.

42 ENGINEERING↗

Software Dependability

The purpose of this paper is to provide a brief introduction of Software Dependability and a summary of lessons learned. The components of Software Dependability vary among standards, however there is agreement among the standards that reliability, availability and maintainability are some the main components of dependability. Safety is also a main component is safety systems. Although reliability and safety are similar there is an additional difference, safety can withstand some level of fault tolerance. For the purposes of this discussion Software Dependability includes reliability, availability, maintainability and safety. Software Dependability: Trustworthiness of a computer system such that reliance can be justifiably placed on the service it delivers. Reliability, availability, and maintainability are aspects of dependability. Reliability: The ability of a system or component to perform its required functions under stated conditions for a specified period of time. Availability: The degree to which a system or component is operational and accessible when required for use. Often expressed as a probability. Maintainability: Speed and ease with which a program can be corrected or changed. Safety: a measure of the length of time the software system can be expected to run before failing in such a way as to cause a system hazard. (author)

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

A user's view of CARE III

The present computerized reliability predictor for digital fault-tolerant systems whose sizes are of the order of one million Markovian equivalent states, employs advanced stochastic modeling techniques and implements a mixed Markov model that enables it to drastically reduce the state size of hitherto computationally unobtainable models. Attention is given to the concepts of failure, fault, and error, in the context of the novel system's fault/error-handling models. Examples are drawn from the system's user-friendly interface dialog.

Bavuso, S. J.↗

The Dangers of Failure Masking in Fault-Tolerant Software: Aspects of a Recent In-Flight Upset Event

On 1 August 2005, a Boeing Company 777-200 aircraft, operating on an international passenger flight from Australia to Malaysia, was involved in a significant upset event while flying on autopilot. The Australian Transport Safety Bureau's investigation into the event discovered that an anomaly existed in the component software hierarchy that allowed inputs from a known faulty accelerometer to be processed by the air data inertial reference unit (ADIRU) and used by the primary flight computer, autopilot and other aircraft systems. This anomaly had existed in original ADIRU software, and had not been detected in the testing and certification process for the unit. This paper describes the software aspects of the incident in detail, and suggests possible implications concerning complex, safety-critical, fault-tolerant software.

Johnson, C. W.↗