Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “failure recovery”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 37 records · Page 2
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Enhanced Component Performance Study: Emergency Diesel Generators 1998–2018

This report presents an enhanced performance evaluation of emergency diesel generators (EDGs) at U.S. commercial nuclear power plants. This report evaluates component performance over time using (1) Institute of Nuclear Power Operations (INPO) Consolidated Events Database (ICES) data from 1998 through 2018 and (2) maintenance unavailability (UA) performance data from Mitigating Systems Performance Index (MSPI) Basis Document data from 2002 through 2018. The objective is to show estimates of current failure probabilities and rates related to EDGs, trend these data on an annual basis, determine if the current data are consistent with the probability distributions currently recommended for use in NRC probabilistic risk assessments, show how the reliability data differ for different EDG manufacturers and for EDGs with different ratings; and summarize the subcomponents, causes, detection methods, and recovery associated with each EDG failure mode. Engineering analyses were performed with respect to time period and failure mode without regard to the actual number of EDGs at each plant. The factors analyzed are: sub-component, failure cause, detection method, recovery, manufacturer, and EDG rating. A statistically significant increasing trend was identified in the frequency of FTLR demands for emergency power system (EPS) and high pressure core spray (HPCS) EDGs and a statistically significant decreasing trend was identified in the frequency of run > 1H hours for EPS and HPCS EDGs.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2020

Holistic Measurement Driven Resilience: Combining Operational Fault and Failure Measurements and Fault Injection for Quantifying Fault Detection, Propagation and Impact. Final report

For HPC systems to date, application resilience to faults and failures has been accomplished by the brute- force method of checkpoint/restart, which allows an application to make forward progress in the face of system and application faults, errors, and failures independent of root cause or end result. It has remained the primary resilience mechanism because we lack a way to identify faults and anticipate consequences early enough to take meaningful mitigating action. However, checkpoint/restart implementations put a tremendous burden on system resources and on the applications themselves and is becoming less feasible at scale. Because we have not yet operated at scales at which checkpoint/restart fails to provide forward progress, despite increasing costs, vendors have had little motivation to provide the instrumentation necessary for early identification of faults and failures. However, as we move from petascale to exascale, component mean time to failure (MTTF) will render the existing techniques ineffectual and/or too expensive. Furthermore, fault recovery mechanisms such as failover and/or error correction introduce performance inconsistency. Instrumentation allowing early indication of problems and tools to enable use of such information by systems, operating systems, and applications offer an alternative, more scalable and less costly solution. In the HMDR project, we built on our experience and expertise developed and accumulated over years of research on design, monitoring, measurement, and assessment of resilient computing systems. Analysis of field data on the current and past generations of extreme-scale systems revealed several challenges that, if not addressed in increasingly larger and more complex systems, may hinder the effectiveness of future exascale computing systems. Specifically, i) file systems and interconnects in current-generation large-scale systems already operate at the margins of resiliency, including consistent performance, and may not scale to larger deployments; ii) automated, software-based failover mechanisms are frequently inadequate and can introduce wider failures, such that failures during recovery may lead to system/application failures, including system-wide outages; and iii) silent data corruption represents a critical fault mode and will require efficient detection mechanisms if next-generation applications are to take full advantage of exascale hardware. To address the above challenges, we assembled a team of world-renowned experts in resilient extreme- scale computing from the University of Illinois (Electrical and Computer Engineering, Computer Science, and NCSA), SNL, LANL, NERSC, and Cray. Our team includes representatives from centers that house many of the largest HPC resources in the world, both today and over the coming years. The team has a unique track record of research in i) system and application failure characterization based on the analysis of field data, ii) data-driven design of fault/error detection mechanisms, and iii) experimental characterization of system/application resiliency. The team includes system owners/operators who provide continuous data collection and access and ensure installation of appropriate analysis tools.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Probing the evolution of fault properties during the seismic cycle with deep learning

We use seismic waves that pass through the hypocentral region of the 2016 M6.5 Norcia earthquake together with Deep Learning (DL) to distinguish between foreshocks, aftershocks and time-to-failure (TTF). Binary and N-class models defined by TTF correctly identify seismograms in test with > 90% accuracy. We use raw seismic records as input to a 7 layer CNN model to perform the classification. Here we show that DL models successfully distinguish seismic waves pre/post mainshock in accord with lab and theoretical expectations of progressive changes in crack density prior to abrupt change at failure and gradual postseismic recovery. Performance is lower for band-pass filtered seismograms (below 10 Hz) suggesting that DL models learn from the evolution of subtle changes in elastic wave attenuation. Tests to verify that our results indeed provide a proxy for fault properties included DL models trained with the wrong mainshock time and those using seismic waves far from the Norcia mainshock; both show degraded performance. Our results demonstrate that DL models have the potential to track the evolution of fault zone properties during the seismic cycle. If this result is generalizable it could improve earthquake early warning and seismic hazard analysis.

58 GEOSCIENCES↗
DOE OSTI2025

EVALUATION OF HRA METHODOLOGIES FOR APPLICATION IN SDP WORK

This study critically evaluates human reliability analysis (HRA) methodologies applicable to regulatory probabilistic safety assessment (PSA) model, with a particular focus on their role in supporting the significance determination process (SDP) in nuclear safety assessment. Firstly, three widely utilized HRA methods – IDHEAS-ECA, SPAR-H, and ASEP/THERP – were qualitatively and quantitatively assessed. Qualitative assessments were conducted using attributes from the NEA/CSNI/R(2015)1 report, while quantitative evaluations employed regression and correlation analyses to compare predicted human error probabilities (HEPs) against empirical data. Results reveal distinct strengths, for example, IDHEAS-ECA’s robust predictive accuracy and K-HRA’s alignment with operational practices. In addition, dependency analysis and recovery analysis were critically evaluated. For dependency analysis, the methods’ handling of inter-task dependencies and their impact on HEPs were examined, while recovery analysis highlighted strategies for mitigating failure events. Furthermore, strategies were proposed to evaluate performance-shaping factors under conditions of reduced human performance, such as stress, fatigue, or cognitive overload, addressing specific challenges faced in SDP evaluations. Human errors from KINS’s operational performance information system event reports were evaluated as a case study. This study identifies gaps and provides actionable insights to ensure their validity and applicability in SDP HRA applications. This paper is a part of research conducted by KINS, and it should be noted that this result does not represent the regulatory position of KINS.

99 - GENERAL AND MISCELLANEOUS↗
DOE OSTI2023

Seamless Transition of Critical Infrastructures using Droop Controlled Grid-forming Inverters

Seamless recovery of power to critical infrastructures, after grid failure, is a crucial need arising in scenarios that are increasingly becoming more frequent. Here, this article proposes a seamless transition strategy using a single and unified mode-dependent droop controlled grid-forming inverters. The control strategy achieves the following objectives: 1) regulates the output active and reactive power by the droop-controlled inverters to a desired value while operating in on-grid mode; 2) seamless transition and recovery of power injections into the load after grid failure by inverters that operates in grid-forming mode all the time; 3) requires only a single bit of information on the grid/network status for the mode transition. A framework for assessing the stability of the system and to guide the choice of parameters for controllers is developed using control-oriented modeling. A controller hardware-in-the-loop-based real-time simulation study on a test system based on the realistic electrical network of a commercial-scale medical center is conducted for initial prototyping of the control strategy. A hardware experiment is conducted with two 3 - $\phi$, 480 -V, 125 -kVA grid-forming inverters, a 3 - $\phi$, 480 -V, 270 -kVA grid simulator, a physical grid switch, and a physical load bank. The experimental data establishes the effectiveness of the always grid-forming operation and control of inverters in meeting power delivery objectives when on-grid and off-grid under various kinds of loads and scenarios while minimizing transients during transitions. Furthermore, performance comparison with existing strategies showcases the advantage of the proposed strategy.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2026

Automated, reliable, and efficient continental-scale replication of 7.3 petabytes of computational simulation data: A case study

We report on our experiences replicating 7.3 petabytes (PB) of Earth System Grid Federation (ESGF) computational simulation data from Lawrence Livermore National Laboratory (LLNL) in California to Argonne National Laboratory (ANL) in Illinois and Oak Ridge National Laboratory (ORNL) in Tennessee—a task motivated by a need for increased reliability, capacity, and performance. This task presented significant challenges: the need to move 29 million files twice under time pressure from aging storage hardware; a source file system bottleneck limiting throughput to 1.5 GB/s; frequent site maintenance windows; and the need for complete reliability at scale. We addressed these challenges using a simple replication tool that invoked Globus to transfer large bundles of files while tracking progress in a database, dynamically rerouting transfers to work around maintenance periods and file system limitations. Under the covers, Globus organized transfers to make efficient use of the high-speed Energy Sciences network (ESnet) and the data transfer nodes deployed at participating sites, and also addressed security, integrity checking, and recovery from a variety of transient failures. This success demonstrates the considerable benefits that can accrue from the adoption of performant data replication infrastructure. The replication tool is available at https://github.com/esgf2-us/data-replication-tools.

Globus↗
DOE OSTI2024

Performance Losses and Current-Driven Recovery from Cation Contaminants in PEM Water Electrolysis

Water contaminants are a common cause of failure for polymer electrolyte membrane (PEM) electrolyzers in the field as well as a confounding factor in research on cell performance and durability. In this study, we investigated the performance impacts of feed water containing representative tap water cations at concentrations ranging from 0.5–500 μ M, with conductivities spanning from ASTM Type II to tap-water levels. We present multiple diagnostic signatures to help identify the presence of contaminants in PEM electrolysis cells. Through analysis of polarization curves and impedance spectroscopy to understand the origins of performance losses, we found that a switch from the acidic to alkaline hydrogen evolution mechanism is a key factor in contaminated cell behavior. Finally, we demonstrated that this mechanism switching can be harnessed to remove cation contaminants and recover cell performance without the use of an acid wash. We demonstrated near-complete recovery of cells contaminated with sodium and calcium, and partial recovery of a cell contaminated with iron, which was further investigated by post-mortem microscopy. The improved understanding of contaminant impacts from this work can inform development of strategies to mitigate or recover performance losses as well as improve the consistency and rigor of electrolysis research.

30 DIRECT ENERGY CONVERSION↗
DOE OSTI2020

Microstructural Changes to Thermally Sprayed Materials Subjected to Dynamic Compression

Dynamic compression of materials can induce a variety of microstructural changes. As thermally-sprayed materials have highly complex microstructures, the expected pressure at which changes occur cannot be predicted a priori. In addition, typical in-situ measurements such as velocimetry are unable to adequately diagnose microstructural changes such as failure or pore collapse. Quasi-isentropic compression experiments with sample recovery were conducted to examine microstructural changes in thermally sprayed tantalum and tantalum-niobium blends up to 8 GPa pressure. Spall fracture was observed in all tests, and post-shot pore volume decreased relative to the initial state. The blended material exhibited larger spall planes with fracture occurring at interphase boundaries. An estimate of the pressure at which pore collapse is complete was determined to be ~26 GPa for pure tantalum and ~19 GPa for the tantalumniobium blend under these loading conditions.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Non-destructive electrochemical diagnosis of failure mechanisms in aqueous zinc batteries

The early detection of secondary reactions that affect the life and performance of zinc manganese oxide batteries requires a shift from conventional time-consuming and often destructive procedures to rapid lifetime-predictive techniques. In this work, an electrochemical approach is employed to elucidate independent signatures for four common types of failure mechanisms in zinc manganese dioxide (Zn||MnO2) batteries—namely, the loss of zinc inventory, the loss of active material at the cathode, electrolyte depletion, and increased cell impedance. Our findings, specific to coin cell configurations, reveal that each induced failure mechanism can be distinctively modeled and identified based on responses from the rest voltage and columbic-efficiency data for prompt detection. For instance, electrolyte depletion response manifests a distinctive abrupt (>80 %) decrease in columbic efficiency (CE) and charge-rest voltage (Vc) while the discharge-rest voltage remained constant at ~1.3 V. Furthermore, electrolyte rejuvenation of the cell increased the CE to >95 % and restored Vc from ~0.3 to >1.7 V. Recovery experiments and reference performance tests demonstrated consistency between electrochemical descriptors and their associated failure mechanisms. Further, the outcomes of this work provide valuable insights and data models for some of the dominant failure mechanisms present in zinc manganese battery chemistries, which are beneficial to accelerated early-lifetime diagnosis and advancement of Zn batteries development.

25 ENERGY STORAGE↗
DOE OSTI2024

Source term analysis of FeCrAl accident tolerant fuel using MELCOR

It has been established that incremental improvements in beyond design basis accident performance can be achieved through accident tolerant fuel (ATF). However, they have the potential to recover margin with respect to conventional fuel and therefore enhance plant economics through uprate or cycle length increase. To realize this potential, it is necessary to quantify the reduction in source term due to use of ATF, and correspondingly how this is affected by increasing the cycle length and/or burnup. This requires development of a risk informed analysis methodology for ATF under high burnup conditions, which is being developed within LWRS. To this end, a MELCOR simulation of FeCrAl ATF in a recovered Large Break LOCA (LBLOCA) scenario has been developed, using a model based on the Zion Pressurized Water Reactor (PWR). The new user defined material capability, along with the inclusion of detailed neutronics- and depletion-derived parameters such as core power distribution, decay heat behavior, and fission product inventories, allows a more detailed simulation of FeCrAl clad material properties and behavior than has previously been possible using MELCOR. These detailed FeCrAl results were compared with a zircaloy clad model to investigate the differences between the two clad types and quantify the benefits of FeCrAl ATF with an 18-month cycle. Next, the fuel cycle with ATF was extended to 24 months, to determine whether any additional safety margin provided by FeCrAl ATF could be leveraged to implement high burnup FeCrAl-clad fuel while retaining the same operating and safety limits as current zircaloy-clad fuel. For the particular scenario analyzed, the delay to fuel failure from using ATF was of the same order as the LOCA recovery time, and hence significant in reducing fission product release, with the 24-month FeCrAl core performing better than the 18-month zircaloy core. It is noted that for other transients, the reduction in release due to using FeCrAl may be less significant. Furthermore, the material model developed here can be used in such further studies in support of determining the overall source term.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗
DOE OSTI2020

Resilience of Urban Transport Network-of-Networks under Intense Flood Hazards Exacerbated by Targeted Attacks

Natural hazards including floods can trigger catastrophic failures in interdependent urban transport network-of-networks (NoNs). Population growth has enhanced transportation demand while urbanization and climate change have intensified urban floods. However, despite the clear need to develop actionable insights for improving the resilience of critical urban lifelines, the theory and methods remain underdeveloped. Furthermore, as infrastructure systems become more intelligent, security experts point to the growing threat of targeted cyber-physical attacks during natural hazards. Here we develop a hypothesis-driven resilience framework for urban transport NoNs, which we demonstrate on the London Rail Network (LRN). We find that topological attributes designed for maximizing efficiency rather than robustness render the network more vulnerable to compound natural-targeted disruptions including cascading failures. Our results suggest that an organizing principle for post-disruption recovery may be developed with network science principles. Our findings and frameworks can generalize to urban lifelines and more generally to real-world spatial networks.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Impact Testing and Analysis of Modified Metal Slugs

This report summarizes initial results from a series of gun experiments which were conducted at the DICE facility. The target of these experiments was a modified metal slug composed of a tantalum/tungsten alloy (Ta-10W). The general geometry of the slug was a right circular cylinder with a through-hole cut normal to the cylinder's axis. In all experiments, hardened steel impactors were used, the desired impact velocity was 200 m/s, the slug was preheated to a target temperature of 175° C, photon doppler velocimetry (PDV) was used to measure the projectile velocity before and after impact, and the impact event was recorded with high-speed video. In two of the impacts the slug was oriented perpendicular to the projectile, while in the remaining two it was tilted 8° from normal. Initial high-speed speed video results showed slug failure in the tilted impact case, while the slug survived normal impacts. Recovery fixtures were used to preserve impacted slugs for future postmortem analysis. Discussions are included regarding improvements to potential future experiments involving these slugs.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2020

Leveraging Flexible Smart Manufacturing to Accelerate Industrial Supply Chain Recovery

Any major crisis such as the latest coronavirus disease pandemic will have a monumental effect on the worldwide economy and on the international supply chain. Much of the global supply chain relies on China, Germany, and the United States for manufacturing and distribution; this fragile system is subject to failure. Climactic changes in demand and rapid shortages in necessities are the results of the pandemic’s disruption. The recovery forecast for the ongoing pandemic is uncertain and, therefore, any recovery effort will need to be adaptable to build the supply chain resiliency. A wide adoption of flexible smart automated technologies in the manufacturing sector are helping to build robust supply chains and assisting in recovery. Here, a brief discussion of these smart manufacturing technologies is presented, along with a list of potential supply chain issues and corresponding solutions. Such smart manufacturing technologies possess the potential to empower and revolutionize the traditional manufacturing environment by enhancing its resiliency and flexibility. These smart technologies will play a crucial role in accelerating the worldwide supply chain recovery in and after a pandemic.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2022

Towards Low-Overhead Resilience for Data Parallel Deep Learning

Data parallel techniques have been widely adopted both in academia and industry as a tool to enable scalable training of deep learning models. At scale, DL training jobs can fail due to software or hardware bugs, may need to be preempted or terminated due to unexpected events, or may perform suboptimally because they were misconfigured. Under such circumstances, there is a need to recover and/or reconfigure data-parallel DL training jobs on-the-fly, while minimizing the impact on the accuracy of the DNN model and the runtime overhead. In this regard, state-of-art techniques adopted by the HPC community mostly rely on checkpoint-restart, which inevitably leads to loss of progress, thus increasing the runtime overhead. In this paper we explore alternative techniques that exploit the properties of modern deep learning frameworks (overlapping of gradient averaging and weight updates with local gradient computations through pipeline parallelism) to reduce the overhead of resilience/elasticity. To this end we introduce a failure simulation framework and two resilience strategies (immediate mini-batch rollback and lossy forward recovery), which we study compared with checkpoint-restart approaches in a variety of settings in order to understand the trade-offs between the accuracy loss of the DNN model and the runtime overhead.

data-parallel training↗
DOE OSTI2025

Resilient Entanglement Distribution in a Multihop Quantum Network

The evolution of quantum networking requires architectures capable of dynamically reconfigurable entanglement distribution to meet diverse user needs and ensure tolerance against transmission disruptions. We introduce multihop quantum networks to improve network reach and resilience by enabling quantum communications across intermediate nodes, thus broadening network connectivity and increasing scalability. We present multihop two-qubit polarization-entanglement distribution within a quantum network at the Oak Ridge National Laboratory campus. Our system uses wavelength-selective switches for adaptive bandwidth management on a software-defined quantum network that integrates a quantum data plane with classical data and control planes, creating a flexible, reconfigurable mesh. Our network distributes entanglement across six nodes within three subnetworks, each located in a separate building, optimizing quantum state fidelity and transmission rate through adaptive resource management. Additionally, we demonstrate the network's resilience by implementing a link recovery approach that monitors and reroutes quantum resources to maintain service continuity despite link failures—paving the way for scalable and reliable quantum networking infrastructures.

Alshowkan, Muneer [Oak Ridge National Laboratory (↗
DOE OSTI2022

Stress-Controlled Creep–Fatigue of an Advanced Austenitic Stainless Steel at Elevated Temperatures

Creep–fatigue interaction occurs in many structural components of high-temperature systems operating under cyclic and steady-state service conditions, such as in nuclear power plants, aerospace, naval, and other industrial applications. Thus, understanding micromechanisms governing high-temperature creep–fatigue behavior is essential for safety and design considerations. In this work, stress-controlled creep–fatigue tests of advanced austenitic stainless steel (Alloy 709) were performed at a 400 MPa stress range and 750 °C with tensile hold times of 0, 60, 600, 1800, and 3600 s, followed by microstructural examinations. The creep–fatigue lifetime of the Alloy 709 was found to decrease with increasing hold time until reaching a saturation level where the number of cycles to failure did not exhibit a significant decrease. Softening behavior was observed at the beginning of the test, possibly due to the recovery of entangled dislocations and de-twining. In addition, hysteresis loops showed ratcheting behavior, although the mean stress was zero during creep–fatigue cycling, which was attributed to activity of partial dislocations. Microstructural examination of the fracture surfaces showed that fatigue failure dominated at small hold times where the cracks initiated at the surface of the sample. Larger creep cracks were found for longer hold times with a lower probability of dimpled cavities, indicating the dominance of creep deformation. The results were compared with other commonly used stainless steels, and plausible reasons for the observed responses were described.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

An Improved Transmission Switching Algorithm for Managing Post-(N-1) Contingencies in Electricity Networks

In this presentation, we detail the shortcomings of existing transmission switching (TS) methodologies for preventing post-contingency loss of load in wide area electric power networks. Following that, we present a bi-level algorithm as an improvement to the state of the art. We show proof-of-concept level results that indicate computational viability, fast solutions, and the ability to pick the same best candidate as non computationally viable methods. Further, our method also minimizes load shedding after the contingency, thus enhancing the reliability and security of electricity supply. We also include a futuristic scenario of increased renewables and decreased natural gas sourced generators to show promising results. We grant permission for our presentation to be recorded and uploaded by the conference organizers.

24 POWER TRANSMISSION AND DISTRIBUTION↗
DOE OSTI2021

Risk-averse optimization for resilience enhancement of complex engineering systems under uncertainties

With the growth of complexity and extent, large scale interconnected network systems, e.g., transportation networks or infrastructure networks, become more vulnerable to external disturbances. Hence, managing potential disruptive events during the design, operating, and recovery phase of an engineered system and therefore improving the system’s resilience is an important yet challenging task. Here, to ensure system resilience after the occurrence of failure events, this study proposes a mixed-integer linear programming (MILP) based restoration framework using heterogeneous dispatchable agents. The scenario-based stochastic optimization (SO) technique is adopted to deal with the inherent uncertainties imposed on the recovery process from nature. Moreover, different from conventional SO using deterministic equivalent formulations, the CVaR risk measure is implemented for this study because of the temporal sparsity of the decision making in applications such as the recovery from extreme events. The resulting restoration framework involves a large-scale MILP problem and thus an adequate decomposition technique i.e. modified Lagrangian dual decomposition, is also employed to achieve tractable computational complexity. Case study results based on the IEEE 37-bus test feeder demonstrate the benefits of using the proposed framework for resilience improvement as well as the advantages of adopting SO formulations.

42 ENGINEERING↗