Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Fault Injection”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Characterizing Impacts of Storage Faults on HPC Applications: A methodology and insights

In recent years, the increasing complexity in scientific simulations and emerging demands for training heavy artificial intelligence models require massive and fast data accesses, which urges high-performance computing (HPC) platforms to equip with more advanced storage infrastructures such as solid-state disks (SSDs). While SSDs offer high-performance I/O, it remains unclear about the reliability challenges faced by the HPC applications under the SSD-related failures, in particular, failures resulting in data corruptions. The goal of this paper is to understand the impact of SSD-related data corruptions on the behaviors of complex HPC applications. To this end, we propose FFIS, a FUSE-based fault injection framework that systematically introduces storage faults into the application layer to model the errors originated from SSDs. FFIS is able to plant different I/O related faults into the data returned from underlying file systems, which also enables the investigation on the error resilience characteristics of the scientific file format for the first time. We demonstrate the use of FFIS with three representative real HPC applications, show how each application reacts to the data corruptions, and provide insights on the error resilience of the widely-adopted HDF5 file format for the HPC applications.

Fang, Bo↗

Estimating the distribution of fault latency in a digital processor

Presented is a statistical approach to measuring fault latency in a digital processor. The method relies on the use of physical fault injection where the duration of the fault injection can be controlled. Although a specific fault's latency period is never directly measured, the method indirectly determines the distribution of fault latency.

Ellis, Erik L.↗

In-circuit fault injector user's guide

A fault injector system, called an in-circuit injector, was designed and developed to facilitate fault injection experiments performed at NASA-Langley's Avionics Integration Research Lab (AIRLAB). The in-circuit fault injector (ICFI) allows fault injections to be performed on electronic systems without special test features, e.g., sockets. The system supports stuck-at-zero, stuck-at-one, and transient fault models. The ICFI system is interfaced to a VAX-11/750 minicomputer. An interface program has been developed in the VAX. The computer code required to access the interface program is presented. Also presented is the connection procedure to be followed to connect the ICFI system to a circuit under test and the ICFI front panel controls which allow manual control of fault injections.

Padilla, Peter A.↗

Determining the Impact of Steady-State PV Fault Current Injections on Distribution Protection

This report investigates the fault current contribution from a single large PV system and the impact it has on existing distribution overcurrent protection devices. Assumptions are made about the modeling of the PV system under fault to perform exhaustive steady - state fault analyses throughout distribution feeder models. Each PV interconnection location is tested to determine how the size of the PV system affects the fault current measured by each protection device. This data is then searched for logical conditions that indicate whether a protection device has operated in a manner that will cause more customer outages due to the addition of the PV system. This is referred to as a protection issue , and there are four unique types of issues that have been identified in the study. The PV system size at which any issues occur are recorded to determine the feeder's PV hosting capacity limitations due to interference with protection settings. The analysis is carried out on six feeder models. The report concludes with a discussion of the prevalence and cause of each protection issue caused by PV system fault current.

30 DIRECT ENERGY CONVERSION↗

Experimental evaluation of the certification-trail method

Certification trails are a recently introduced and promising approach to fault-detection and fault-tolerance. A comprehensive attempt to assess experimentally the performance and overall value of the method is reported. The method is applied to algorithms for the following problems: huffman tree, shortest path, minimum spanning tree, sorting, and convex hull. Our results reveal many cases in which an approach using certification-trails allows for significantly faster overall program execution time than a basic time redundancy-approach. Algorithms for the answer-validation problem for abstract data types were also examined. This kind of problem provides a basis for applying the certification-trail method to wide classes of algorithms. Answer-validation solutions for two types of priority queues were implemented and analyzed. In both cases, the algorithm which performs answer-validation is substantially faster than the original algorithm for computing the answer. Next, a probabilistic model and analysis which enables comparison between the certification-trail method and the time-redundancy approach were presented. The analysis reveals some substantial and sometimes surprising advantages for ther certification-trail method. Finally, the work our group performed on the design and implementation of fault injection testbeds for experimental analysis of the certification trail technique is discussed. This work employs two distinct methodologies, software fault injection (modification of instruction, data, and stack segments of programs on a Sun Sparcstation ELC and on an IBM 386 PC) and hardware fault injection (control, address, and data lines of a Motorola MC68000-based target system pulsed at logical zero/one values). Our results indicate the viability of the certification trail technique. It is also believed that the tools developed provide a solid base for additional exploration.

Sullivan, Gregory F.↗

Modeling injection-induced fault slip using long short-term memory networks

Stress changes due to changes in fluid pressure and temperature in a faulted formation may lead to the opening/shearing of the fault. This can be due to subsurface (geo)engineering activities such as fluid injections and geologic disposal of nuclear waste. Such activities are expected to rise in the future making it necessary to assess their short- and long-term safety. Here, a new machine learning (ML) approach to model pore pressure and fault displacements in response to high-pressure fluid injection cycles is developed. The focus is on fault behavior near the injection borehole. To capture the temporal dependencies in the data, long short-term memory (LSTM) networks are utilized. To prevent error accumulation within the forecast window, four critical measures to train a robust LSTM model for predicting fault response are highlighted: (i) setting an appropriate value of LSTM lag, (ii) calibrating the LSTM cell dimension, (iii) learning rate reduction during weight optimization, and (iv) not adopting an independent injection cycle as a validation set. Several numerical experiments were conducted, which demonstrated that the ML model can capture peaks in pressure and associated fault displacement that accompany an increase in fluid injection. The model also captured the decay in pressure and displacement during the injection shut-in period. Further, the ability of an ML model to highlight key changes in fault hydromechanical activation processes was investigated, which shows that ML can be used to monitor risk of fault activation and leakage during high pressure fluid injections.

58 GEOSCIENCES↗

Dynamic simulation of CO 2 -injection-induced fault rupture with slip-rate dependent friction coefficient

It is known that poro-elastic stress and effective stress reduction associated with deep underground fluid injection can potentially trigger shear rupture along pre-existing faults. We modeled an idealized CO 2 injection scenario, to assess the effects on faults in the first phase of a generic CO 2 aquifer storage operation. We used coupled multiphase fluid flow and geomechanical numerical modeling to evaluate the stress and pressure perturbations induced by fluid injection and the response of a nearby normal fault. Slip-rate dependent friction and inertial effects have been taken into account during rupture. Contact elements have been used to take into account the frictional behavior of the rupture plane. We investigated different scenarios of injection rate to induce rupture on the fault, employing various fault rheologies. Published laboratory data on CO 2 -saturated intact and crushed rock samples, representative of a potential target aquifer, sealing formation and fault gouge, have been used to define a scenario where different fault rheologies apply at different depths. Nucleation of fault rupture takes place at the bottom of the reservoir, in agreement with analytical poro-elastic stress calculations, depending on injection-induced reservoir inflation and the tectonic stress scenario. For the stress state considered here, the first triggered rupture always produces the largest rupture length and slip magnitude, both of which correlate with the fault rheology. Velocity weakening produces larger ruptures and generates larger magnitude seismic events. Heterogeneous faults have been considered including velocity-weakening or velocity strengthening sections inside and below the aquifer, with the upper sections being velocity-neutral. Nucleation of rupture in a velocity-strengthening section results in a limited rupture extension, both in terms of maximum slip and rupture length. For a heterogeneous fault with nucleation in a velocity-weakening section, the rupture may propagate into the overlying velocity-neutral section, if the extent of velocity-weakening and associated friction drop are large enough.

58 GEOSCIENCES↗

Fail-Safe Logic Design Strategies Within Modern FPGA Architectures

Fail-safe computing refers to computing systems that revert to a non-operational safe state when a fault occurs. In this paper, we investigate a circuit level technique as mitigation for single event upsets (SEUs) and fault injection attacks on field programmable gate arrays (FPGAs), and analyze the effectiveness of the technique as a fail-safe monitor for an encryption algorithm. The propagation of fault effects through FPGA primitives including lookup tables (LUTs) and programmable interconnect points (PIPs) is assessed within an FPGA architecture created using an open source tool, and validated using fault injection experiments on an FPGA. The analysis reveals additional vulnerabilities exist within reconfigurable architectures over those in equivalent fail-safe application specific integrated circuit (ASIC), thus requiring a more elaborate network of redundant circuits and checking logic. The configuration memory bits (CMBs), which configure routing and designate logic functions within the LUTs of the FPGA, add complexity to fail-safe design strategies by introducing additional fault conditions and fault propagation paths. A resource-efficient fail-safe circuit design technique called DEsign for Fail-safe in reCONfigurable systems (DEFCON) is proposed. The benefits and limitations associated with DEFCON are described in the context of fault injection experiments carried out as simulations and in FPGA hardware.

Bhakta, Priya A. [Univ. of New Mexico, Albuquerque↗

Establishing Fault Tolerance for a Class of Systems by Experiment

A long-standing problem in system verification is establishing fault tolerance at the ultra-high level by experiment. It is considered impossible because of system complexity and the enormous number of trials needed. This paper considers the problem for a class of digital systems that use redundancy to achieve reliability. The class is the systems that operate for a period of time without maintenance followed by a maintenance check that replaces components identified as faulty. The paper considers simulating a natural life test where a natural life test observes a number of operating periods. If the system does not fail during the test, it can be said to have a certain reliability at a certain confidence level. The approach in this paper is to make the simulated life test more efficient while maintaining realism by integrating structural arguments, information on fault occurrence, and fault injection in the lab. The major result of this paper is constructing a global fault model using the failure rate of the components and proving theorems about the model that tell how many, what kind, when, and where to inject faults. A simple example illustrates applying the theorems.

design of experiments↗

Characterization of the Impact of Soft Errors on Iterative Methods

Soft errors caused by transient bit flips have the potential to significantly impact an application’s behavior. This has motivated the design of an array of techniques to detect, isolate, and correct soft errors using microarchitectural, architectural, compilation-based, or application-level techniques to minimize their impact on the executing application. The first step towards the design of good error detection/correction techniques involves an understanding of an application’s vulnerability to soft errors. In this paper, we present the first comprehensive characterization of the impact of soft errors on the convergence characteristics of six iterative methods using application-level fault injection. In particular, we consider the use of iterative methods to incrementally solve a linear systems of equations, which constitutes the core kernel in many scientific applications. We analyze the impact of soft errors in terms of the type of error (single- vs multi-bit), the distribution and location of bits affected, the data structure and the statement impacted, and variation with time. In addition to understanding the vulnerability of iterative solvers to soft errors, this characterization can aid the design of fault injection campaigns that ensure systematic coverage.

fault injection, iterative methods, Silent data co↗

Development of an Automated Software Reliability Tester for Digital I and C

Existing nuclear facilities are in the process of upgrading analog systems to new, software- based digital components; however, the qualification of these new systems has been a significant hurdle for their implementation in the United States. This paper describes the development of a Software Reliability Tester (SRT) that will provide a means to quantify the reliability and fault tolerance of digital instrumentation and control (I and C) systems used in nuclear facilities. The SRT is comprised of hardware and software that exercise digital equipment inputs and compare the outputs to expected values to quantify system reliability. Incorporation of fault injection techniques enables the SRT to evaluate the robustness of digital I and C in the presence of abnormal conditions. The integration of both quantitative reliability and fault tolerance measures will provide the nuclear industry with an automated and systematic approach to digital I and C qualification. This will enable safer, more reliable, and more cost effective implementations of a broad spectrum of digital I and C equipment in existing and next generation nuclear facilities. This paper includes a description of the design of the SRT, including software modules and the hardware interface. In addition, an example of using the SRT to automate functional testing of a digital timer relay is presented. The SRT described in this paper provides a means to conduct efficient and robust testing for a variety of digital devices. Future plans for the SRT include integrating fault injection testing so that devices can be evaluated in the presence of off-normal conditions. (authors)

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

The Gremlin Framework

The Gremlin sofrware package is a performance analysis approach targeted to support the Co-Design process for future systems. It consists of a series of modules that can be used to alter a machine's behavior with the goal of emulating future machine properties. The modules can be divided into several classes; the most significant ones are detailed below. PowGre is a series of modules that help explore the power consumption properties of applications and to determine the impact of power constraints on applications. Most of them use low-level processor interfaces to directly control voltage and frequency settings as well as per nodes, socket, or memory power bounds. MemGre are memory Gremlins and implement a new performance analysis technique that captures the application's effective use of the storage capacity of different levels of the memory hierarchy as well as the bandwidth between adjacent levels. The approach models various memory components as resources and measures how much of each resource the application uses from the application's own perspective. To the application a given amount of a resource is "used" if not having this amount will degrade the application's performance. This is in contrast to the hardware-centric perspective that considers "use" as any hardware action that utilizes the resource, even if it has no effect on performance. ResGre are Gremlins that use fault injection techniques to emulate higher fault rates than currently present in today's systems. Faults can be injected through various means, including network interposition, static analysis, and code modification, or direct application notification. ResGre also includes patches to previously released LLNL codes that can counteract and react to injected failures.

Source record↗

Hierarchical Simulation to Assess Hardware and Software Dependability

This thesis presents a method for conducting hierarchical simulations to assess system hardware and software dependability. The method is intended to model embedded microprocessor systems. A key contribution of the thesis is the idea of using fault dictionaries to propagate fault effects upward from the level of abstraction where a fault model is assumed to the system level where the ultimate impact of the fault is observed. A second important contribution is the analysis of the software behavior under faults as well as the hardware behavior. The simulation method is demonstrated and validated in four case studies analyzing Myrinet, a commercial, high-speed networking system. One key result from the case studies shows that the simulation method predicts the same fault impact 87.5% of the time as is obtained by similar fault injections into a real Myrinet system. Reasons for the remaining discrepancy are examined in the thesis. A second key result shows the reduction in the number of simulations needed due to the fault dictionary method. In one case study, 500 faults were injected at the chip level, but only 255 propagated to the system level. Of these 255 faults, 110 shared identical fault dictionary entries at the system level and so did not need to be resimulated. The necessary number of system-level simulations was therefore reduced from 500 to 145. Finally, the case studies show how the simulation method can be used to improve the dependability of the target system. The simulation analysis was used to add recovery to the target software for the most common fault propagation mechanisms that would cause the software to hang. After the modification, the number of hangs was reduced by 60% for fault injections into the real system.

Ries, Gregory Lawrence↗

A Runtime Environment for Supporting Research in Resilient HPC System Software & Tools

The high-performance computing (HPC) community continues to increase the size and complexity of hardware platforms that support advanced scientific workloads. The runtime environment (RTE) is a crucial layer in the software stack for these large-scale systems. The RTE manages the interface between the operating system and the application running in parallel on the machine. The deployment of applications and tools on large-scale HPC computing systems requires the RTE to manage process creation in a scalable manner, support sparse connectivity, and provide fault tolerance. We have developed a new RTE that provides a basis for building distributed execution environments and developing tools for HPC to aid research in system software and resilience. This paper describes the software architecture of the Scalable runTime Component Infrastructure (STCI), which is intended to provide a complete infrastructure for scalable start-up and management of many processes in large-scale HPC systems. We highlight features of the current implementation, which is provided as a system library that allows developers to easily use and integrate STCI in their tools and/or applications. The motivation for this work has been to support ongoing research activities in fault-tolerance for large-scale systems. We discuss the advantages of the modular framework employed and describe two use cases that demonstrate its capabilities: (i) an alternate runtime for a Message Passing Interface (MPI) stack, and (ii) a distributed control and communication substrate for a fault-injection tool.

Computer Science↗

Model-Based Verification and Validation of Spacecraft Avionics

Verification and Validation (V&V) at JPL is traditionally performed on flight or flight-like hardware running flight software. For some time, the complexity of avionics has increased exponentially while the time allocated for system integration and associated V&V testing has remained fixed. There is an increasing need to perform comprehensive system level V&V using modeling and simulation, and to use scarce hardware testing time to validate models; the norm for thermal and structural V&V for some time. Our approach extends model-based V&V to electronics and software through functional and structural models implemented in SysML. We develop component models of electronics and software that are validated by comparison with test results from actual equipment. The models are then simulated enabling a more complete set of test cases than possible on flight hardware. SysML simulations provide access and control of internal nodes that may not be available in physical systems. This is particularly helpful in testing fault protection behaviors when injecting faults is either not possible or potentially damaging to the hardware. We can also model both hardware and software behaviors in SysML, which allows us to simulate hardware and software interactions. With an integrated model and simulation capability we can evaluate the hardware and software interactions and identify problems sooner. The primary missing piece is validating SysML model correctness against hardware; this experiment demonstrated such an approach is possible.

MBV&V↗

Modeling of fault activation and seismicity by injection directly into a fault zone associated with hydraulic fracturing of shale-gas reservoirs

We conducted three-dimensional coupled fluid-flow and geomechanical modeling of fault activation and seismicity associated with hydraulic fracturing stimulation of a shale-gas reservoir. We simulated a case in which a horizontal injection well intersects a steeply dip- ping fault, with hydraulic fracturing channeled within the fault, during a 3-hour hydraulic fracturing stage. Consistent with field observations, the simulation results show that shale-gas hydraulic fracturing along faults does not likely induce seismic events that could be felt on the ground surface, but rather results in numerous small microseismic events, as well as aseismic deformations along with the fracture propagation. The calculated seismic moment magnitudes ranged from about -2.0 to 0.5, except for one case assuming a very brittle fault with low residual shear strength, for which the magnitude was 2.3, an event that would likely go unnoticed or might be barely felt by humans at its epicenter. The calculated moment magnitudes showed a dependency on injection depth and fault dip. We attribute such dependency to variation in shear stress on the fault plane and associated variation in stress drop upon reactivation. Our simulations showed that at the end of the 3-hour injection, the rupture zone associated with tensile and shear failure extended to a maximum radius of about 200 m from the injection well. The results of this modeling study for steeply dipping faults at 1000 to 2500 m depth is in agreement with earlier studies and field observations showing that it is very unlikely that activation of a fault by shale-gas hydraulic fracturing at great depth (thousands of meters) could cause felt seismicity or create a new flow path (through fault rupture) that could reach shallow groundwater resources.

54 ENVIRONMENTAL SCIENCES↗