Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Fault Injection”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Mathematical Modelling of Fault Reactivation Induced by Water Injection

Faults in the host rock that might exist in the vicinity of deep geological repositories for radioactive waste, constitute potential enhanced pathways for radionuclide migration. Several processes might trigger pore pressure increases in the faults leading to fault failure and induced seismicity, and increase the faults’ permeability. In this research, we developed a mathematical model to simulate fault activation during an experiment of controlled water injection in a fault at the Mont-Terri Underground Research Laboratory in Switzerland. The effects of in-situ stress, fault shear strength parameters and heterogeneity are assessed. It was shown that the above factors are critical and need to be adequately characterized in order to predict the faults’ hydro-mechanical behaviour.

58 GEOSCIENCES↗

Experimental analysis of computer system dependability

This paper reviews an area which has evolved over the past 15 years: experimental analysis of computer system dependability. Methodologies and advances are discussed for three basic approaches used in the area: simulated fault injection, physical fault injection, and measurement-based analysis. The three approaches are suited, respectively, to dependability evaluation in the three phases of a system's life: design phase, prototype phase, and operational phase. Before the discussion of these phases, several statistical techniques used in the area are introduced. For each phase, a classification of research methods or study topics is outlined, followed by discussion of these methods or topics as well as representative studies. The statistical techniques introduced include the estimation of parameters and confidence intervals, probability distribution characterization, and several multivariate analysis methods. Importance sampling, a statistical technique used to accelerate Monte Carlo simulation, is also introduced. The discussion of simulated fault injection covers electrical-level, logic-level, and function-level fault injection methods as well as representative simulation environments such as FOCUS and DEPEND. The discussion of physical fault injection covers hardware, software, and radiation fault injection methods as well as several software and hybrid tools including FIAT, FERARI, HYBRID, and FINE. The discussion of measurement-based analysis covers measurement and data processing techniques, basic error characterization, dependency analysis, Markov reward modeling, software-dependability, and fault diagnosis. The discussion involves several important issues studies in the area, including fault models, fast simulation techniques, workload/failure dependency, correlated failures, and software fault tolerance.

Iyer, Ravishankar, K.↗

Understanding Mixed Precision GEMM with MPGemmFI: Insights into Fault Resilience

Emerging deep learning workloads urgently need fast general matrix multiplication (GEMM). Thus, one of the critical features of machine-learning-specific accelerators such as NVIDIA Tensor Cores, AMD Matrix Cores, and Google TPUs is the support of mixed-precision enabled GEMM. For DNN models, lower-precision FP data formats and computation offer acceptable correctness but significant performance, area, and memory footprint improvement. While promising, the mixed-precision computation on error resilience remains unexplored. To this end, we develop a fault injection framework that systematically injects fault into the mixed-precision computation results. We investigate how the faults affect the accuracy of machine learning applications. Based on the characteristics of error resilience, we offer lightweight error detection and correction solutions that significantly improve the overall model accuracy by 75% if the models experience hardware faults. The solutions can be efficiently integrated into the accelerator's pipelines.

Fang, Bo↗

Recent results from a testbed for the reliability of the digital instrumentation and control systems and cybersecurity in nuclear power plants - 392

Transition from analog to digital instrumentation and control systems requires a systematic study of the modifications, and assessment of their reliability. Such studies are often carried out on a test bed. The test bed developed earlier includes a commercially available triple modular redundant (TMR) programmable logic controller (PLC) for testing not only the reliability of the digital I and C but also the cyber-security of the system during plant operation. This paper reports results of our preliminary analyses carried out using the test bed following the injection of faults mimicking faulty sensors. The fault injection tool generates faulty signals in single or multiple sensors located in the primary loop. These scenarios studied under steady-state operation conditions simulate a bad sensor, a cut wire, noise, etc. The faults injected can lead to information being displayed in the main control room (MCR) to not match the conditions in the plant. In addition to a fault injection module, a result analyzer module is also a part of this testbed. Operator actions can be based on information displayed in the MCR, or they can be informed by the result analyzer. Potential operator actions following these single- or multiple faulty signals are assessed. It is concluded that the result analyzer in some cases can be helpful in guiding operators to make the right decisions. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Characterizing the Potential for Injection-Induced Fault Reactivation Through Subsurface Structural Mapping and Stress Field Analysis, Wellington Field, Sumner County, Kansas: Injection-Induced Fault Slip, Kansas

Kansas, like other parts of the central U.S., has experienced a recent increase in seismicity. Correlation of these events with brine disposal operations suggests pore fluid pressure increases are reactivating preexisting faults, but rigorous evaluation at injection sites is lacking. Here we determine the suitability of CO 2 injection into the Cambrian-Ordovician Arbuckle Group for long-term storage and into a Mississippian reservoir for enhanced oil recovery in Wellington Field, Sumner County, Kansas. To determine the potential for injection-induced earthquakes, we map subsurface faults and estimate in situ stresses, perform slip and dilation tendency analyses to identify well-oriented faults relative to the estimated stress field, and determine the pressure changes required to induce slip at reservoir and basement depths. Three-dimensional seismic reflection data reveal 12 near-vertical faults, mostly striking NNE, consistent with nodal planes from moment tensor solutions from recent earthquakes in the region. Most of the faults cut both reservoirs and several clearly penetrate the Precambrian basement. Drilling-induced fractures (N = 40) identified from image logs and inversion of earthquake moment tensor solutions (N = 65) indicate that the maximum horizontal stress is approximately EW. Slip tendency analysis indicates that faults striking <020° are stable under current reservoir conditions, whereas faults striking 020°–049° may be prone to reactivation with increasing pore fluid pressure. Although the proposed injection volume (40,000 t) is unlikely to reactive faults at reservoir depths, high-rate injection operations could reach pressures beyond the critical threshold for slip within the basement, as demonstrated by the large number of injection-induced earthquakes west of the study area.

54 ENVIRONMENTAL SCIENCES↗

Control Mechanisms for Self‐Sealing in Activated Clay‐Rich Faults Through Controlled Hydraulic Injection Experiment

Abstract In a high‐pressure injection fault activation experiment conducted at the Mont Terri underground research laboratory in Switzerland, the transmissivity of the Opalinus Clay fault significantly increased due to opening and shearing. The fluid injection, spanning a few hours, generated a 10 m radius fault activation patch. Subsequent pressure pulse tests conducted bi‐weekly for a year revealed the gradual return of fault transmissivity to its initial state. The study utilized fluid pressure decay analysis, optical fiber monitoring, continuous active source seismic measurements and borehole displacement sensors for measuring fault displacements. The fault zone exhibited a dilation of approximately 1.4 mm, associated with both normal and tangential movements during activation, resulting in a sudden transmissivity increase from 1 × 10 −12 to 3.2 × 10 −7 m 2 /s. Early post‐activation, transient compaction and the subsequent slow compaction were observed, transitioning to an extension regime. The pressure pulse tests demonstrated a rapid transmissivity drop by more than two orders of magnitude within the first 10 days, followed by a gradual and less pronounced decrease. Plastic shear and compaction dominated the transmissivity evolution until 70 days after injection ended, followed by a period where additional factors, such as clay mineral swelling, influenced the behavior. Extrapolation suggested a sealing process taking at least 50 years after the initial activation. Plain Language Summary A field‐scale fault activation experiment offers valuable insights into the elasto‐plastic processes governing the sealing of shale faults. The experiment reveals a rapid increase in the fault's transmissivity by approximately five orders of magnitude during activation. Subsequent observations show a gradual transmissivity decrease by about three orders of magnitude post‐activation, with slow long‐term plastic shear and compaction of the fault competing against secondary processes, notably clay mineral swelling. All conceptual models employed to interpret these field data converge on the estimation that the fault's return to its initial low transmissivity state would require a minimum of 50 years. Key Points High‐pressure injection fault activation experiment at the Mont Terri underground research laboratory Continuous transmissivity measurements record self‐sealing inside a clay‐rich fault zone Transmissivity undergoes a phase of domination by slow plastic compaction and shearing during the initial post‐activation period, with mineral swelling exerting its influence over the long term

Guglielmi, Yves↗

Modeling of fluid injection-induced fault reactivation using coupled fluid flow and mechanical interface model

The present study is aimed at developing a numerical model to reproduce coupled hydro-mechanical processes associated with fault reactivation by fluid injection in low permeability rock, as part of the DECOVALEX-2019 project Task B. We proposed a modeling approach for simulating the processes using the TOUGH-FLAC simulator, and modeled a fault reactivation experiment conducted at Mont Terri Rock Laboratory in Switzerland. The first step of the study involved benchmark calculations considering a simplified fault plane and geometry. Fluid flow along a fault was modeled using elements of aperture-sized thickness on the basis of Darcy's law and the cubic law in TOUGH2, whereas the mechanical behavior of a single fault was represented by zero-thickness interface elements in FLAC3D upon which a slip and/or separation is allowed. A methodology to connect a TOUGH2 volume element to a FLAC3D interface element was developed for handling the hydro-mechanical interactions on the fault during fluid injection. Two different fault models for describing the evolutions of hydraulic aperture by elastic fracture opening and failure-induced aperture increase were considered in the benchmark calculations. In the coupling process, the changes in geometrical features and hydrological properties induced by mechanical deformation were continuously updated. The transient responses of the fault and host rock to stepwise pressurization were examined during the simulation. The hydro-mechanical behavior, including the injection flow rate, pressure distribution around the borehole, stress conditions, and displacements in normal and shear directions were monitored in the surrounding rock and along the fault. The results of benchmark calculations suggest that the developed model reasonably represents the hydro-mechanical behavior of a fault and the surrounding rock. This modeling approach was applied to the fault reactivation experiment of the Mont Terri Rock Laboratory. In this interpretive modeling, a parametric study was conducted to examine the effects of input parameters regarding in situ stress and fault properties on the hydro-mechanical responses of the fault to water injection. Then, an optimal parameter set to reproduce the field experiment results was chosen by trial-and-error. The injection flow rate and pressure response during fault reactivation closely matched those obtained at the site, which indicates the capability of the model to appropriately capture the progressive pathway evolution during fault reactivation tests at the site. Finally, the anchor displacements were overestimated by the model, but a fair agreement was obtained in terms of the order of magnitude and the variation tendency.

58 GEOSCIENCES↗

Fault recovery characteristics of the fault tolerant multi-processor

The fault handling performance of the fault tolerant multiprocessor (FTMP) was investigated. Fault handling errors detected during fault injection experiments were characterized. In these fault injection experiments, the FTMP disabled a working unit instead of the faulted unit once every 500 faults, on the average. System design weaknesses allow active faults to exercise a part of the fault management software that handles byzantine or lying faults. It is pointed out that these weak areas in the FTMP's design increase the probability that, for any hardware fault, a good LRU (line replaceable unit) is mistakenly disabled by the fault management software. It is concluded that fault injection can help detect and analyze the behavior of a system in the ultra-reliable regime. Although fault injection testing cannot be exhaustive, it has been demonstrated that it provides a unique capability to unmask problems and to characterize the behavior of a fault-tolerant system.

Padilla, Peter A.↗

ARETE: Accurate Error Assessment via Machine Learning-Guided Dynamic-Timing Analysis

Nanometer circuits are increasingly prone to timing errors, escalating the need for fault injection frameworks to accurately evaluate their impact on applications. Here in this paper, we propose ARETE, a novel cross-layer, fault-injection framework that combines dynamic-binary instrumentation with machine learning-guided dynamic-timing analysis. ARETE enables accurate fault-injection into any application by estimating the location of the injecting errors via dynamic-timing analysis. To accelerate fault-injection, we develop a novel, data-aware, machine learning-based mechanism that dynamically pre-selects the error-prone instructions and limits the application of the costly dynamic-timing analysis only to them. To evaluate ARETE's accuracy, our fully automated toolflow is configured to support fault-injection based on detailed post-layout gate-level simulations as well as via existing workload-agnostic error models. Our results for various workloads, including an autonomous-driving library, show that the location and time of injected errors performed by ARETE, is 89.9% consistent with fault-injection based on full gate-level simulation. On average, ARETE executes 84.6x faster than gate-level simulation and at a cost of 3.4% loss in the program output quality estimation. When compared to the existing statistical fault-injection tools that are based on workload-agnostic error models, ARETE improves the accuracy of fault-injection rate and output quality estimation by 143.9% and 40.4% on average, respectively.

97 MATHEMATICS AND COMPUTING↗

Dynamic modeling of injection-induced fault reactivation and ground motion and impact on surface structures and human perception

We summarize recent modeling studies of injection-induced fault reactivation, seismicity, and its potential impact on surface structures and nuisance to the local human population. We used coupled multiphase fluid flow and geomechanical numerical modeling, dynamic wave propagation modeling, seismology theories, and empirical vibration criteria from mining and construction industries. We first simulated injection-induced fault reactivation, including dynamic fault slip, seismic source, wave propagation, and ground vibrations. From co-seismic average shear displacement and rupture area, we determined the moment magnitude to about M w = 3 for an injection-induced fault reactivation at a depth of about 1000 m. We then analyzed the ground vibration results in terms of peak ground acceleration (PGA), peak ground velocity (PGV), and frequency content, with comparison to the U.S. Bureau of Mines’ vibration criteria for cosmetic damage to buildings, as well as human-perception vibration limits. For the considered synthetic M w = 3 event, our analysis showed that the short duration, high frequency ground motion may not cause any significant damage to surface structures, and would not cause, in this particular case, upward CO 2 leakage, but would certainly be felt by the local population.

58 GEOSCIENCES↗

SpotSDC: Revealing the Silent Data Corruption Propagation in High-Performance Computing Systems

We report the trend of rapid technology scaling is expected to make the hardware of high-performance computing (HPC) systems more susceptible to computational errors due to random bit flips. Some bit flips may cause a program to crash or have a minimal effect on the output, but others may lead to silent data corruption (SDC), i.e., undetected yet significant output errors. Classical fault injection analysis methods employ uniform sampling of random bit flips during program execution to derive a statistical resiliency profile. However, summarizing such fault injection result with sufficient detail is difficult, and understanding the behavior of the fault-corrupted program is still a challenge. In this article, we introduce SpotSDC, a visualization system to facilitate the analysis of a program's resilience to SDC. SpotSDC provides multiple perspectives at various levels of detail of the impact on the output relative to where in the source code the flipped bit occurs, which bit is flipped, and when during the execution it happens. SpotSDC also enables users to study the code protection and provide new insights to understand the behavior of a fault-injected program. Based on lessons learned, we demonstrate how what we found can improve the fault injection campaign method.

97 MATHEMATICS AND COMPUTING↗

Understanding scale-dependent soft-error behavior of scientific applications

Analyzing application fault behavior on large-scale systems is time-consuming and resource-demanding. Currently, researchers need to perform fault injection campaigns at full scale to understand the effects of soft errors on applications and whether these faults result in silent data corruption. Both time and resource requirements greatly limit the scope of the resilience studies that can be currently performed. In this work, we propose a methodology to model application fault behavior at large scale based on a reduced set of experiments performed at small scale. We employ machine learning techniques to accurately model application fault behavior using a set of experiments that can be executed in parallel at small scale. Our methodology drastically reduces the set and the scale of the fault injection experiments to be performed and provides a validated methodology to study application fault behavior at large scale. We show that our methodology can accurately model application fault behavior at large scale by using only small scale experiments. In some cases, we can model the fault behavior of a parallel application running on 4,096 cores with about 90% accuracy based on experiments on a single core.

Kestor, Gokcen G.↗

Board Level Proton Testing Book of Knowledge for NASA Electronic Parts and Packaging Program

This book of knowledge (BoK) provides a critical review of the benefits and difficulties associated with using proton irradiation as a means of exploring the radiation hardness of commercial-off-the-shelf (COTS) systems. This work was developed for the NASA Electronic Parts and Packaging (NEPP) Board Level Testing for the COTS task. The fundamental findings of this BoK are the following. The board-level test method can reduce the worst case estimate for a board's single-event effect (SEE) sensitivity compared to the case of no test data, but only by a factor of ten. The estimated worst case rate of failure for untested boards is about 0.1 SEE/board-day. By employing the use of protons with energies near or above 200 MeV, this rate can be safely reduced to 0.01 SEE/board-day, with only those SEEs with deep charge collection mechanisms rising this high. For general SEEs, such as static random-access memory (SRAM) upsets, single-event transients (SETs), single-event gate ruptures (SEGRs), and similar cases where the relevant charge collection depth is less than 10 μm, the worst case rate for SEE is below 0.001 SEE/board-day. Note that these bounds assume that no SEEs are observed during testing. When SEEs are observed during testing, the board-level test method can establish a reliable event rate in some orbits, though all established rates will be at or above 0.001 SEE/board-day. The board-level test approach we explore has picked up support as a radiation hardness assurance technique over the last twenty years. The approach originally was used to provide a very limited verification of the suitability of low cost assemblies to be used in the very benign environment of the International Space Station (ISS), in limited reliability applications. Recently the method has been gaining popularity as a way to establish a minimum level of SEE performance of systems that require somewhat higher reliability performance than previous applications. This sort of application of the method suggests a critical analysis of the method is in order. This is also of current consideration because the primary facility used for this type of work, the Indiana University Cyclotron Facility (IUCF) (also known as the Integrated Science and Technology (ISAT) hall), has closed permanently, and the future selection of alternate test facilities is critically important. This document reviews the main theoretical work on proton testing of assemblies over the last twenty years. It augments this with review of reported data generated from the method and other data that applies to the limitations of the proton board-level test approach. When protons are incident on a system for test they can produce spallation reactions. From these reactions, secondary particles with linear energy transfers (LETs) significantly higher than the incident protons can be produced. These secondary particles, together with the protons, can simulate a subset of the space environment for particles capable of inducing single event effects (SEEs). The proton board-level test approach has been used to bound SEE rates, establishing a maximum possible SEE rate that a test article may exhibit in space. This bound is not particularly useful in many cases because the bound is quite loose. We discuss the established limit that the proton board-level test approach leaves us with. The remaining possible SEE rates may be as high as one per ten years for most devices. The situation is actually more problematic for many SEE types with deep charge collection. In cases with these SEEs, the limits set by the proton board-level test can be on the order of one per 100 days. Because of the limited nature of the bounds established by proton testing alone, it is possible that tested devices will have actual SEE sensitivity that is very low (e.g., fewer than one event in 1 × 10(exp 4) years), but the test method will only be able to establish the limits indicated above. This BoK further examines other benefits of proton board-level testing besides hardness assurance. The primary alternate use is the injection of errors. Error injection, or fault injection, is something that is often done in a simulation environment. But the proton beam has the benefit of injecting the majority of actual SEEs without risk of something being missed, and without the risk of simulation artifacts misleading the SEE investigation.

Guertin, Steven M.↗