Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “fault injection sampling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

Characterization of fault recovery through fault injection on FTMP

The development of fault-injection procedures and statistical analysis techniques to characterize the fault recovery of fault-tolerant systems is described. Pin-level fault-injection was conducted on a fault-tolerant microprocessor computer in order to generate data to assess the utility of current fault-injection sampling methods. The validity of common reliability-modeling assumptions concerning the statistical distribution of recovery times is investigated. A multiple comparison analysis for detecting behavior variations, and a distribution fitting for determining the best fit for the data were conducted. It is observed that the detection behavior is not homogeneous across all data sets, and that none of the factors under experimental control can account for the observed groupings of behavior. It is determined that no single distribution fits all the data sets, and that stratified random sampling and statistically robust parameter-estimation techniques are required to characterize fault detection time.

Finelli, George B.↗

Error latency measurements in symbolic architectures

Error latency, the time that elapses between the occurrence of an error and its detection, has a significant effect on reliability. In computer systems, failure rates can be elevated during a burst of system activity due to increased detection of latent errors. A hybrid monitoring environment is developed to measure the error latency distribution of errors occurring in main memory. The objective of this study is to develop a methodology for gauging the dependability of individual data categories within a real-time application. The hybrid monitoring technique is novel in that it selects and categorizes a specific subset of the available blocks of memory to monitor. The precise times of reads and writes are collected, so no actual faults need be injected. Unlike previous monitoring studies that rely on a periodic sampling approach or on statistical approximation, this new approach permits continuous monitoring of referencing activity and precise measurement of error latency.

Young, L. T.↗

Experimental analysis of computer system dependability

This paper reviews an area which has evolved over the past 15 years: experimental analysis of computer system dependability. Methodologies and advances are discussed for three basic approaches used in the area: simulated fault injection, physical fault injection, and measurement-based analysis. The three approaches are suited, respectively, to dependability evaluation in the three phases of a system's life: design phase, prototype phase, and operational phase. Before the discussion of these phases, several statistical techniques used in the area are introduced. For each phase, a classification of research methods or study topics is outlined, followed by discussion of these methods or topics as well as representative studies. The statistical techniques introduced include the estimation of parameters and confidence intervals, probability distribution characterization, and several multivariate analysis methods. Importance sampling, a statistical technique used to accelerate Monte Carlo simulation, is also introduced. The discussion of simulated fault injection covers electrical-level, logic-level, and function-level fault injection methods as well as representative simulation environments such as FOCUS and DEPEND. The discussion of physical fault injection covers hardware, software, and radiation fault injection methods as well as several software and hybrid tools including FIAT, FERARI, HYBRID, and FINE. The discussion of measurement-based analysis covers measurement and data processing techniques, basic error characterization, dependency analysis, Markov reward modeling, software-dependability, and fault diagnosis. The discussion involves several important issues studies in the area, including fault models, fast simulation techniques, workload/failure dependency, correlated failures, and software fault tolerance.

Iyer, Ravishankar, K.↗

Uncovering Hazards Using a Multi-Objective Optimization to Explore the Faulty State-Space

Considering resilience when designing complex engineered systems is crucial to ensure the system is safe under unexpected hazardous scenarios. Traditional risk-based approaches, such as Failure Modes and Effects Analysis (FMEA) are useful for designing the system to mitigate hazardous scenarios that can be identified by the designer, but often require experience or prior knowledge of system failures to generate. More recently, researchers have developed simulation tools that enable the designer to model large sets of hazardous scenarios (driven by both internal faults and external factors) through simulation. While these tools enable a wider scope of fault modes to be evaluated (e.g., by injecting combined set of fault modes or injecting modes at different times), the resulting assessments (like FMEA) still require knowledge of the specific modes to be evaluated. However, failure to analyze a wide variety of fault scenarios can lead to an incomplete picture of the system resilience, especially to "surprise events'' which may be difficult for the designer to identify and predict beforehand. To overcome this challenge, previous work developed a fault sampling approach for resilience simulations which would procedurally-generate a wide variety of potential faults by systematically perturbing the health states of the system. While the resulting fault modes generated covered a much larger space hazards than would be otherwise considered (and identified many unique failure trajectories which would not have otherwise been identified), it also significantly increased the computational cost of the analysis and resulted in the simulation and analysis of a large set of essentially duplicate scenarios. Additionally, as the number of dimensions in the faulty state-space increases, the full elaboration of possible modes becomes computationally infeasible, justifying the use of a more targeted search. To resolve this limitation, this work proposes the use of a multiobjective optimization algorithm to search the health state space for potential fault modes that are both (1) hazardous and (2) unique. To solve this type of problem, this work proposes the use of a cooperative co-evolutionary algorithm. To demonstrate this approach, it will be applied to a model of an autonomous rover which uses line markings to navigate, focusing on potential hazards in the drive system which could cause the rover to crash. To determine the merit of the approach, it will further be compared with the previously-presented range elaboration approach and a random mode generation approach on the basis of computational efficiency and found modes.

Resilience↗

Some methods of estimating a coverage parameter

To demonstrate the high reliability of fault-tolerant computing systems, experimental testing can be performed by injecting faults into their hardware and observing the times needed to detect and isolate failed components and reconfigure the good components. Coverage, a parameter measuring continued system success, is the probability that recovery occurs before other, perhaps catastrophic, component failures occur. Methods of estimating coverage are given for the case of randomly selecting a subset of the pins or chips for testing. Pin-level samples of times to recovery are treated by the Type I censoring model often used in life testing.

Lee, L. D.↗

Synthetic Failure Mode Generation for Resilience Analysis and Failure Mechanism Discovery

Traditional risk-based design processes seek to mitigate operational hazards by manually identifying possible faults and corresponding mitigation strategies—a tedious process which critically relies on the designer’s limited knowledge. Resilience-based design, on the other hand, seeks to embody generic hazard-mitigating properties in the system to mitigate unknown hazards, often by modelling the system's response to potential hazardous events. This work adapts this approach to the traditional risk-based design process to synthetically generate hazardous modes, by representing them as a unique combination of internal component health-states which can then be injected and simulated in a model of the system failure dynamics. The design process may then reduce the risk of unknown internal hazards by iteratively mitigating the effects of these modes. The performance of this approach is evaluated in a model of an autonomous rover, where cluster analysis shows that elaborating the space of synthetic faults in the drive system using this approach uncovers a wider range of possible hazardous trajectories and failure consequences within each trajectory. However, this increase in hazard information comes at a high computational expense, highlighting the need for advanced, efficient methods to search and sample the hazard space.

Simulation↗

Synthetic Failure Mode Generation for Resilience Analysis and Failure Mechanism Discovery

Traditional risk-based design processes seek to mitigate operational hazards by manually identifying possible faults and corresponding mitigation strategies—a tedious process which critically relies on the designer’s limited knowledge. Resilience-based design, on the other hand, seeks to embody generic hazard-mitigating properties in the system to mitigate unknown hazards, often by modelling the system's response to potential hazardous events. This work adapts this approach to the traditional risk-based design process to synthetically generate hazardous modes, by representing them as a unique combination of internal component health-states which can then be injected and simulated in a model of the system failure dynamics. The design process may then reduce the risk of unknown internal hazards by iteratively mitigating the effects of these modes. The performance of this approach is evaluated in a model of an autonomous rover, where cluster analysis shows that elaborating the space of synthetic faults in the drive system using this approach uncovers a wider range of possible hazardous trajectories and failure consequences within each trajectory. However, this increase in hazard information comes at a high computational expense, highlighting the need for advanced, efficient methods to search and sample the hazard space.

Simulation↗

Autonomous Ocean World Exploration: Advancement of a Virtual Testbed

The search for life (extinct or extant) and potentially habitable bodies in our solar system and beyond is one of the 12 priority science questions outlined in the National Acadamies’ 2022 decadal survey [5]. Extraterrestrial destinations containing liquid water present an opportunity to search for life as we know it, and in recent years an increasing number of such locations have been discovered within our solar system. Several Jovian moons—Europa, Ganymede, and Callisto [10]—and the Saturnian moons Enceladus [8] and Titan [9] are known or suspected to harbor massive subsurface oceans. Of these "ocean worlds", Europa is the focus of at least one planned NASA orbiter mission, Europa Clipper [4], and an early lander mission concept, the Europa Lander [2, 3]. Whereas most robotic missions to the Moon and Mars (e.g. orbiters, rovers, landers) to date have had ground controllers on Earth tightly involved in mission operations, missions to more distant worlds will require a high degree of onboard autonomy due to long communication lags and blackouts, harsh environments (radiation, cold), and more limited battery and hardware life. The past decade has seen great advances in both AI technologies and computing scalability and performance that offer promising solutions for spacecraft autonomy and motivate the software system and research programs described in this paper. The Ocean Worlds Autonomy Testbed for Exploration, Research, and Simulation (OceanWATERS) [1], which has been in development at the NASA Ames Research Center since 2018, is a virtual environment for testing lander autonomy solutions. It is built on the Robot Operating System (ROS), runs on consumer-grade Linux workstations, and was released as open source in 2020. OceanWATERS provides a physical and visual simulation of a prototypical lander in a Europa-like environment (Figure 1). The lander was modeled after requirements and specifications made in JPL’s Europa Lander Study of 2016 [3]. Simulated lander systems include stereo cameras and spotlights mounted on an antenna mast that pans and tilts, a 6 degrees of freedom (DoF) robotic arm with a force-torque sensor and two interchangeable end effectors, and a battery pack power system. The environment consists of multiple terrain models including a highly detailed model sourced from the FROST dataset [11], simulation of surrounding planetary bodies based on an ephemeris model, and lighting from the sun with associated surface illumination, reflectance, and shadows. Operations supported by OceanWATERS include panoramic and directed imaging of the environment and lander workspace, Cartesian and joint-level arm commanding, grinding of the terrain surface (e.g. digging a trench), and scooping of ground material (Figure 2) which can be discarded or collected as science samples in a receptacle that can be emptied (science operations themselves are not simulated). These operations are realized as ROS Actions and are complimented by a wide selection of telemetry that is continually produced by each lander subsystem. The power system model is driven by the open-source Generic Software Architecture for Prognostics (GSAP) [11] that predicts the battery’s remaining useful life and other characteristics. As a testbed for high-level autonomy, OceanWATERS provides an execution framework based on PLEXIL [12], an open-source plan specification language and execution engine developed largely at Ames. NASA's initial development of OceanWATERS, as well the Ocean Worlds Lander Autonomy Testbed (OWLAT) [6], a complimentary physical testbed developed at JPL, was the first step in a plan for realizing candidate onboard autonomy solutions for such planetary landers. In 2020 NASA solicited applications for its Autonomous Robotics Research for Ocean Worlds (ARROW) program, and in 2021 the similar Concepts for Ocean worlds Life Detection Technology (COLDTech) program. Collectively six research teams, based in universities and companies across the United States, were awarded grants to develop and demonstrate autonomy solutions on OceanWATERS and OWLAT. These 1–2-year projects have now finished or are nearing completion, and a wide variety of autonomy challenges in ocean world surface missions were addressed. Prototyped and demonstrated solutions have included autonomous discovery, response and adaptation to system faults and unexpected environmental events, world model synthesis through perception, plan synthesis using learned models, methods to optimize sample target selection and prioritize science data transmission, extension of PLEXIL for stochastic decision-making, and an integration of a model of JPL’s mission-ready COLDArm [7]. Technologies used in these projects include many forms of machine learning, causal reasoning, automated planning, Markov decision processes, formal methods, and other advanced techniques. A more detailed summary of the ARROW and COLDTech projects is given herein. OceanWATERS has had significant enhancements since its open-source release in 2020. Many of its new features were driven or shaped by feedback from the ARROW and COLDTech teams and requirements of their projects. In support of enabling autonomous adaptation to spacecraft faults (a specific capability solicited by both programs), a fault injection and detection framework was developed that supports a wide and growing range of fault types such as locked joints, image loss, and battery failures. The power system model was completed and integrated into the simulator, starting as a single-cell battery model and later upgraded to a multi-cell model with associated faults such as cell disconnection. Arm/terrain interaction was improved by adding a force-torque sensor and associated faults, and an analytic dig force model based on the Balovnev bucket force equations. Environment fidelity was increased by modeling terrain deformation resulting from digging and scooping; visual improvements were made in textures, lighting, and shadows. To facilitate interoperation with OWLAT, a unified command and telemetry interface between the testbeds was developed at the ROS level, along with a PLEXIL interface. The number of lander operations was greatly expanded (e.g. with Cartesian-based arm and antenna movement), and a framework was designed for users to build their own lander actions. A GUI for PLEXIL plan selection was created (Figure 3), and an expansive set of plans were added, such as those that illustrate patterns for fault handling. This paper provides a self-contained high-level description of OceanWATERS, focusing on more detailed coverage of the aforementioned enhancements. It provides a high-level summary of the projects undertaken by participants in the ARROW and COLDTech programs and how these efforts have helped shape OceanWATERS. Finally, potential future work and directions for the testbed are listed, as likely informed by the recent planetary science decadal survey [5].

K Michael Dalal↗

Validation of the Mars 2020 Fault Protection Design: Navigating the Infinity of the Off-Nominal

On July 30th 2020, the Mars 2020 mission successfully launched out of Cape Canaveral, Florida, passed through the Earth’s shadow, and began its short cruise to Mars. Less than seven months later, the Perseverance rover touched down safely in Jezero Crater to begin its ambitious mission that includes looking for signs of ancient life and collecting samples for future return to Earth. Getting to the successful landing, or “Tango Delta Nominal,” could not have been achieved without also considering the off-nominal. One of the teams supporting this ambitious mission is the fault protection (FP) team. This team is tasked with assessing the various failures, or faults, that could prevent mission success and with ensuring that the autonomous behaviors built into the software and hardware can detect faults and recover the vehicle to a safe state. As part of its charter, the FP team designed a test campaign to provide confidence in the system’s robustness to off-nominal scenarios across all of Mars 2020’s mission phases. The greatest challenge associated with designing such a validation campaign was reducing the infinite number of anomalous scenarios into a finite test suite. In addition, the tests needed to be executed efficiently in order to utilize the team’s limited test venue access, but still needed to maintain a level of rigor that guaranteed confidence in the test outcomes. Given that each test scenario generated massive amounts of data, the team also developed methods for quickly ascertaining whether the autonomous fault protection behaviors maintained vehicle safety in the presence of an anomaly. This paper summarizes the processes that the Mars 2020 fault protection team employed to execute its off-nominal validation campaign. It captures both the methods of generating a suite of off-nominal tests, as well as reducing it to a subset that can be realistically executed within schedule and resource constraints. It also describes the various processes and philosophies that the team utilized to execute the tests efficiently, including creating a standardized procedure template, keeping the test cases modular so that they could be easily interchanged, and capturing common fault injections in a change-controlled database. Finally, it will describe the tools and processes for assessing the test data, focusing in particular on a tool that evaluated vehicle state using “secondary” sources of data to validate that the software had truly configured the spacecraft to the expected safe state.

Morantz, Chaz↗