Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “root cause analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Root cause analysis of a molten salt pump in FLUSTFA

The primary salt pump installed in the high-temperature FLUoride Salt Test Facility (FLUSTFA) was successfully operated for some time, but later ceased operation. To understand what occurred, a Root Cause Analysis (RCA) was performed. Steps taken to try to get the pump operational include adjusting the shaft position, increasing the heating power of the tape heaters on the pump volute, and manually rotating the pump shaft. While removing the insulation, corrosion was noted on the outside of the pump volute, and decolorization of the insulation and tape heaters was observed. Significant corrosion products were also observed in the pump itself and the piping connected to the pump. The nitrogen cover gas was maintained from before salt was introduced into the loop until the pump was dismounted and continues to be maintained even after the pump was removed. After considering probable scenarios, causes were assigned and corrective actions were developed to prevent those causes. Then, the RCA was presented to an advisory committee for review, the “Review Committee,” consisting of experts in large molten salt systems: Brandon Haugh, David Holcomb, Kevin Robb, and Vicente Rojas. As a result, the advisory committee provided comprehensive feedback, which have been incorporated into a revised RCA. Findings have then been summarized and reported in this publication.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

Root cause analysis and solutions for plastic gamma detector degradation in challenging environments—An overview

Polyvinyl toluene (PVT) plastic is often used to make gamma ray sensitive scintillators and is used in many important applications. For example, PVT is used to scan recycled steel going into a plant to be processed. It is also used in personnel portals to scan employees, and is used at the borders of many countries to scan cargo and cars passing through. In recent years, it was discovered that PVT can become degraded when it absorbs water and then is subjected to daily temperature swings, as is evidence by a reduction in scintillation light. Water absorption by PVT was notionally associated with temporary fogging (i.e., a rapid decrease in opacity which was reversible over time) and permanent fogging (i.e., permanent crack-like defects) and was for some time suspected to be the leading cause of the fogging due to circumstantial evidence. However, definitive proof was not established and in particular the specific mechanisms by which water entered the plastic and ultimately created temporary and permanent point-like defects was not fully understood. This paper is an overview of an effort initiated to reveal the fundamental root cause and also the dynamics of the fogging and degradation processes. This understanding is important not only to solve degradation problems for PVT in fielded systems, but also (1) to provide direction for future procurements for new equipment, and (2) to provide direction for future R&D efforts into advanced plastics with enhanced spectroscopic capabilities or dual particle gamma/neutron capabilities. Unexpected outcomes of this investigation included (1) the capability to predict the onset of fogging based on models using weather data alone, and (2) new modified formulations of PVT and PS which are fog-resistant. This paper provides an overview of the water uptake and associated mechanisms for various compositions, characterization of resulting temporary and permanent defects, predictive modeling of fogging, and possible solutions including encapsulation, heaters and new formulations. While this paper is an overview of the root cause analysis and solution, five accompanying papers (Lance et al., 2018, Myllenbeck and Feng, 2018, Payne et al., 2018, Zaitseva et al. 2018, Kouzes et al., 2018) from the same conference (SORMA 2018) delve further into the details and also expand upon the scope of what will be reported herein.

36 MATERIALS SCIENCE↗

Root Cause Analysis of Degradation in Protonic Ceramic Electrochemical Cell with Interfacial Electrical Sensors Using Data‐Driven Machine Learning

Abstract Protonic ceramic electrochemical cells (PCECs) offer promising paths for energy storage and conversion. Despite considerable achievements made, PCECs still face challenges such as physiochemical compatibility between componenets and suboptimal solid–solid contact at the interfaces between the electrolytes and electrodes. In this study, a novel approach is proposed that combines in situ electrochemical characterization of interfacial electrical sensor embedded PCECs and machine learning to quantify the contributions of different cell components to total degradation, as well as to predict the remaining useful life. The experimental results suggest that the overpotential induced by the oxygen electrode is 48% less than that of oxygen electrode/electrolyte interfacial contact for up to 1171 h. The data‐driven machine learning simulation predicts the RUL of up to 2132 h. The root cause of degradation is overpotential increase induced by oxygen electrode, which accounts for 82.9% of total cell degradation. The success of the failure diagnostic model is demonstrated by its consistency with degradation modes that do not manifest in electrolysis fade during early real operations. This synergistic approach provides valuable insights into practical failure diagnosis of PCECs and has the potential to revolutionize their development by enabling improved performance prediction and material selection for enhanced durability and efficiency.

25 ENERGY STORAGE↗

Root Cause Analysis of AC Overcurrent in July 2020 San Fernando Disturbance

On July 7 2020, approximately 1,000 MW reduction in solar photovoltaic (PV) output was experienced by the bulk power system in Southern California after a three-phase fault. Disturbance analysis indicates that instantaneous AC current caused inverter tripping. The projection of the root cause of AC overcurrent from the NERC report is that inverter current control is not tight. This letter provides a quantitative analysis on ac overcurrent and addresses the following questions: (i) what type of current controls may lead to ac overcurrent; and (ii) how controller parameters and grid strength influence ac overcurrent. Here, the letter provides frequency-domain analysis and time-domain simulation results of a grid-integrated inverter to illustrate the effect of control parameters and grid strength on ac overcurrent.

14 SOLAR ENERGY↗

Feedback and Oscillations: Constructing Feedback Systems for Root Cause Analysis of Oscillations in Power Grids

Dynamic phenomena linked to inverter-based resources (IBRs) have gained global attention. Several IBR-induced dynamics have caused bulk power system-connected wind or solar power plants to trip, and some have even led to widespread outages. In addition, many oscillations have been observed involving IBR power plants. In 2023, the IEEE Power & Energy Society (PES) IBR Subsynchronous Oscillations (SSO) task force published a journal article, “Real-World Subsynchronous Oscillation Events in Power Grids With High Penetrations of Inverter-Based Resources,” in which 19 IBR oscillation events were examined for their causation. Earlier in 2020, another PES task force article, “Definition and Classification of Power System Stability-Revisited & Extended,” authored by prominent academics, introduced converter-driven stability as a new category of stability. The international power grid industry community also took action by publishing the CIGRE Green Book, Power System Dynamic Modelling and Analysis in Evolving Networks (led by Babak Badrzadeh and Zia Emin) in 2024. In August 2024, the Energy Systems Integration Group (ESIG) released a practical guide led by Nick Miller, “Diagnosis and Mitigation of Observed Oscillations in IBR-Dominant Power System: A Practical Guide.” The goal of the guide is to assist practicing engineers in making initial judgments and conducting detailed analyses about oscillations. Finally, when addressing the classification of stability and oscillations, the guide emphasizes a causality-based taxonomy for grouping, such as voltage control-induced oscillations, synchronization-induced oscillations, and frequency or active power control-induced oscillations.

Fan, Lingling [Univ. of South Florida, Tampa, FL (↗

WISP: Watching grid Infrastructure Stealthily through Proxies (Final Technical Report)

The complex interdependencies of cyber systems (sensors and communications), physical grids and associated electricity market operations make protecting electric power grids a significant challenge. The energy sector is constantly under new, targeted, advanced and dangerous cyber-attacks that have the potential to result in the loss of human life. These threats are further exacerbated by our need to modernize the grid. One focus of cyber security research in smart grids is the securing of the SCADA system through advanced intrusion detection systems (IDS) and bad data detection algorithms in state estimation. These methods either require full knowledge of the system topology and parameters or fail to understand the physical behaviors under attack. WISP (Watching grid Infrastructure Stealthily through Proxies) is designed to provide additional protection to the power grid using only publicly available data. In particular, WISP exploits the spatio-temporal nature of the real time locational marginal prices (LMPs), in conjunction with other information such as bids, weather, outages and load data to analyze anomalous power pricing behaviors and then correlate those observations to localize regions of interest and identify potential cyber events. WISP is non-intrusive as the tool is deployed as a service in the Cloud or on premise and provides reliable information to system operators for enhanced situational awareness, without impeding energy delivery functions. The WISP technology comprises three modules: the data-driven anomaly detection core, the vulnerability and risk analysis and the root cause analysis. The data-driven anomaly detection core performs the tasks of feature selection, anomaly detection and attack region localization. The vulnerability and risk analysis module provides system level information of the vulnerable variables and times, assisting the operators in selecting monitoring and protection nodes. The root cause analysis module takes the detection results and identifies potential operational conditions that contribute to the detected anomalies. In Phase I, we have demonstrated the feasibility and effectiveness of WISP. We developed a realistic electricity market simulator capable of generating normal and attack market data under various operational conditions. We developed a series of cyber-attack detection and analysis algorithms and evaluated them under multiple data sources. Finally, we integrated all modules into an end-to-end software, providing functions for data management, data analytics and visualization. Specifically, we have achieved: (i) real-time data acceptance from external utility interfaces with >99% acceptance rate; (ii) high performance anomaly detection algorithms with >98% detection accuracy and <0.1% false alarm rate; and (iii) ultra-low computing delay <50 milliseconds. Additionally, our team developed algorithms to identify the vulnerable variables in electricity market operations and root cause analysis functions to identify major contributors to the price spikes. These ancillary modules are necessary when deploying WISP in real world industry environment. In Phase II, we have demonstrated the effectiveness of WISP software on realistic largescale power systems. We performed red team testing for the Phase I WISP software and identified software vulnerabilities and implemented corresponding mitigation solutions. We adapted the electricity market simulator for the Texas synthetic 2000-bus system and generated datasets for the false data injection attacks. We created database and visualization interfaces for the Texas system and the ISO New England system. We performed software optimization in terms of operation efficiency, computing speed and detection accuracy. Finally, we tested the software on the Texas system and the ISO New England system and evaluated the detection performance. Overall, we achieved above 89% detection rate, below 3% false alarm rate and below 37 seconds of end-to-end detection delay.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Towards Unlocking Insights from Logbooks Using AI

Electronic logbooks contain valuable information about activities and events concerning their associated particle accelerator facilities. However, the highly technical nature of logbook entries can hinder their usability and automation. As natural language processing (NLP) continues advancing, it offers opportunities to address various challenges that logbooks present. This work explores jointly testing a tailored Retrieval Augmented Generation (RAG) model for enhancing the usability of particle accelerator logbooks at institutes like DESY, BESSY, Fermilab, BNL, SLAC, LBNL and CERN. The RAG model uses a corpus built on logbook contributions and aims to unlock insights from these logbooks by leveraging retrieval over facility datasets, including discussion about potential multimodal sources. Our goals are to increase the FAIR-ness (findability, accessibility, interoperability, and reusability) of logbooks by exploiting their information content to streamline everyday use, to enable macro-analysis for root cause analysis, and to facilitate problem-solving automation.

43 PARTICLE ACCELERATORS↗

Towards Unlocking Insights from Logbooks Using AI

Electronic logbooks contain valuable information about activities and events concerning their associated particle accelerator facilities. However, the highly technical nature of logbook entries can hinder their usability and automation. As natural language processing (NLP) continues advancing, it offers opportunities to address various challenges that logbooks present. This work explores jointly testing a tailored Retrieval Augmented Generation (RAG) model for enhancing the usability of particle accelerator logbooks at institutes like DESY, BESSY, Fermilab, BNL, SLAC, LBNL, and CERN. The RAG model uses a corpus built on logbook contributions and aims to unlock insights from these logbooks by leveraging retrieval over facility datasets, including discussion about potential multimodal sources. Our goals are to increase the FAIR-ness (findability, accessibility, interoperability, and reusability) of logbooks by exploiting their information content to streamline everyday use, enable macro-analysis for root cause analysis, and facilitate problem-solving automation.

43 PARTICLE ACCELERATORS↗

Root Cause Correlation Analysis of Software Failures via Orthogonal Defect Classification and Natural Language Processing

Systems theoretic process analysis (STPA) is becoming an increasingly popular technique to assess how complex digital software systems can fail. Rather than defining failures by their observable failure events, which may be sparse especially for safety rated nuclear digital instrumentation and control systems (DI&C), failures are defined as postulated unsafe actions under specific contextual conditions. This permits a top-down analysis of system hazards and identifies whether imposed constraints and requirements can sufficiently address undesirable hazards. However, STPA is a qualitative approach at identifying inadequacies in the development process and cannot currently be used to quantify unsafe action likelihoods for probabilistic risk assessment. Therefore, in this work, we examine the root causes of software failure and explore whether a consistent correlation can be linked to specific unsafe action classes. We implement Lbl2Vec, an unsupervised document classification and retrieval algorithm, on a database of 4,096 software defect reports acquired from various open-source software systems. By analyzing sentence structure, embedded labels, and word vectors, we show that certain defect types positively correlate to specific unsafe action classes over others. The correlations developed can be used to estimate the failure probability of safety intended DI&C systems which provides a licensing basis for nuclear plant modernization efforts.

22 - GENERAL STUDIES OF NUCLEAR REACTORS↗

Root Cause Correlation Analysis of Software Failures via Orthogonal Defect Classification and Natural Language Processing

Systems theoretic process analysis (STPA) is becoming an increasingly popular technique to assess how complex digital software systems can fail. Rather than defining failures by their observable failure events, which may be sparse especially for safety rated nuclear digital instrumentation and control systems (DI&C), failures are defined as postulated unsafe actions under specific contextual conditions. This permits a top-down analysis of system hazards and identifies whether imposed constraints and requirements can sufficiently address undesirable hazards. However, STPA is a qualitative approach at identifying inadequacies in the development process and cannot currently be used to quantify unsafe action likelihoods for probabilistic risk assessment. Therefore, in this work, we examine the root causes of software failure and explore whether a consistent correlation can be linked to specific unsafe action classes. We implement Lbl2Vec, an unsupervised document classification and retrieval algorithm, on a database of 4,096 software defect reports acquired from various open-source software systems. By analyzing sentence structure, embedded labels, and word vectors, we show that certain defect types positively correlate to specific unsafe action classes over others. The correlations developed can be used to estimate the failure probability of safety intended DI&C systems which provides a licensing basis for nuclear plant modernization efforts.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

MO-F-16A-04: Case Study: Estimation of Peak Skin Dose Following a Physician Reported “High Dose” Case and Sentinel Event Considerations

Purpose: This work offers as a teaching example a reported high dose fluoroscopy case and the workflow the institution followed to self-report a radiation overdose sentinel event to the Joint Commission. Methods: Following the completion of a clinical case in a hybrid OR room with a reported air kerma of >18 Gy at the Interventional Reference Point (IRP) the physicians involved in the case referred study to the institution's Radiation Safety Committee (RSC) for review. The RSC assigned a Diagnostic Medical Physicist (DMP) to estimate the patient's Peak Skin Dose (PSD) and analyze the case. Following the DMP's analysis and estimate of a PSD of >15 Gy the institution's adverse event committee was convened to discuss the case and to self-report the case as a radiation overdose sentinel event to the Joint Commission. The committee assigned a subgroup to perform the root cause analysis and develop institutional responses to the event. Results: The self-reporting of the sentinel event and the associated root cause analysis resulted in several institutional action items that are designed to improve process and safety. A formal reporting and analysis mechanism was adopted to review fluoroscopy cases with air kerma greater than 6 Gy at the IRP. An improved and formalized radiation safety training program for physicians using fluoroscopy equipment was implemented. Additionally efforts already under way to monitor radiation exposure in the Radiology department were expanded to include all fluoroscopy equipment capable of automated dose reporting. Conclusion: The adverse event review process and the root cause analysis following the self-reporting of the sentinel event resulted in policies and procedures that are expected to improve the quality and safe usage of fluoroscopy throughout the institution.

60 APPLIED LIFE SCIENCES↗

Monitoring Extreme-scale Lustre Toolkit

We discuss the design and ongoing development of the Monitoring Extreme-scale Lustre Toolkit (MELT), a unified Lustre performance monitoring and analysis infrastructure that provides continuous, low-overhead summary information on the health and performance of Lustre, as well as on-demand, in-depth problem diagnosis and root-cause analysis. The MELT infrastructure leverages a distributed overlay network to enable monitoring of center-wide Lustre filesystems where clients are located across many network domains. We preview interactive command-line utilities that help administrators and users to observe Lustre performance at various levels of resolution, from individual servers or clients to whole filesystems, including job-level reporting. Finally, we discuss our future plans for automating the root-cause analysis of common Lustre performance problems.

97 MATHEMATICS AND COMPUTING↗

Validation Progress and Plans for ENDF/B-VIII.1β1 at ORNL [Slides]

This presentation opens with a ORNL definition of ENDF/B-VIII.1β1. Following with keff results for CE calculations with VALID. Additionally provided are Preliminary Root Cause Analysis (RCA) results. This presentation then concludes with future plans for additional validation.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

TH-B-BRC-00: How to Identify and Resolve Potential Clinical Errors Before They Impact Patients Treatment: Lessons Learned

Radiation treatment consists of a chain of events influenced by the quality of machine operation, beam data commissioning, machine calibration, patient specific data, simulation, treatment planning, imaging and treatment delivery. There is always a chance that the clinical medical physicist may make or fail to detect an error in one of the events that may impact on the patient’s treatment. In the clinical scenario, errors may be systematic and, without peer review, may have a low detectability because they are not part of routine QA procedures. During treatment, there might be errors on machine that needs attention. External reviews of some of the treatment delivery components by independent reviewers, like IROC, can detect errors, but may not be timely. The goal of this session is to help junior clinical physicists identify potential errors as well as the approach of quality assurance to perform a root cause analysis to find and eliminate an error and to continually monitor for errors. A compilation of potential errors will be presented by examples of the thought process required to spot the error and determine the root cause. Examples may include unusual machine operation, erratic electrometer reading, consistent lower electron output, variation in photon output, body parts inadvertently left in beam, unusual treatment plan, poor normalization, hot spots etc. Awareness of the possibility and detection of error in any link of the treatment process chain will help improve the safe and accurate delivery of radiation to patients. Four experts will discuss how to identify errors in four areas of clinical treatment. D. Followill, NIH grant CA 180803.

60 APPLIED LIFE SCIENCES↗

TH-B-BRC-01: How to Identify and Resolve Potential Clinical Errors

Radiation treatment consists of a chain of events influenced by the quality of machine operation, beam data commissioning, machine calibration, patient specific data, simulation, treatment planning, imaging and treatment delivery. There is always a chance that the clinical medical physicist may make or fail to detect an error in one of the events that may impact on the patient’s treatment. In the clinical scenario, errors may be systematic and, without peer review, may have a low detectability because they are not part of routine QA procedures. During treatment, there might be errors on machine that needs attention. External reviews of some of the treatment delivery components by independent reviewers, like IROC, can detect errors, but may not be timely. The goal of this session is to help junior clinical physicists identify potential errors as well as the approach of quality assurance to perform a root cause analysis to find and eliminate an error and to continually monitor for errors. A compilation of potential errors will be presented by examples of the thought process required to spot the error and determine the root cause. Examples may include unusual machine operation, erratic electrometer reading, consistent lower electron output, variation in photon output, body parts inadvertently left in beam, unusual treatment plan, poor normalization, hot spots etc. Awareness of the possibility and detection of error in any link of the treatment process chain will help improve the safe and accurate delivery of radiation to patients. Four experts will discuss how to identify errors in four areas of clinical treatment. D. Followill, NIH grant CA 180803.

60 APPLIED LIFE SCIENCES↗

TH-B-BRC-02: Troubleshooting Issues with Linear Accelerator

Radiation treatment consists of a chain of events influenced by the quality of machine operation, beam data commissioning, machine calibration, patient specific data, simulation, treatment planning, imaging and treatment delivery. There is always a chance that the clinical medical physicist may make or fail to detect an error in one of the events that may impact on the patient’s treatment. In the clinical scenario, errors may be systematic and, without peer review, may have a low detectability because they are not part of routine QA procedures. During treatment, there might be errors on machine that needs attention. External reviews of some of the treatment delivery components by independent reviewers, like IROC, can detect errors, but may not be timely. The goal of this session is to help junior clinical physicists identify potential errors as well as the approach of quality assurance to perform a root cause analysis to find and eliminate an error and to continually monitor for errors. A compilation of potential errors will be presented by examples of the thought process required to spot the error and determine the root cause. Examples may include unusual machine operation, erratic electrometer reading, consistent lower electron output, variation in photon output, body parts inadvertently left in beam, unusual treatment plan, poor normalization, hot spots etc. Awareness of the possibility and detection of error in any link of the treatment process chain will help improve the safe and accurate delivery of radiation to patients. Four experts will discuss how to identify errors in four areas of clinical treatment. D. Followill, NIH grant CA 180803.

60 APPLIED LIFE SCIENCES↗