Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “root cause analysis”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 73 records · Page 4

Preparation and Qualification of Preproduction SSR2 Jacketed Cavities for PIP-II

The qualification of 325 MHz Single Spoke Resonators type 2 (SSR2) jacketed cavities to meet technical requirements represents a significant milestone in the development of the SSR2 cryomodules for the PIP-II Project at Fermilab. This poster reports the procedures and lessons learned in processing and preparing these cavities for horizontal cold testing prior to integration into a cavity string assembly, with a focus on addressing the field emission issues observed during the cold testing. A comprehensive root cause analysis identified critical fabrication, processing, and handling factors impacting field emission onset. New techniques were successfully developed and implemented to achieve field emission-free SSR2 cavities, and efforts were made to correlate radiation levels measured at the test stand with expected levels in the LINAC tunnel. Additionally, the evolution of field emission through assembly steps was thoroughly investigated, leading to a reassessment of design choices and enhancing our understanding of their effects on cavity performance.

Grassellino, L. [Fermilab]↗

Enhanced Preparation for Intelligent Cybermanufacturing Systems (EPICS)

Opportunities exist for realizing transformative advances in productivity and reductions in energy footprint through ubiquitous sensing in manufacturing environments. Enhanced Preparation for Intelligent Cybermanufacturing Systems (EPICS) is a 21-month (4 academic semesters, plus one summer) experience for graduate students that focuses on scaling the knowledge, understanding and leadership skills in the cyber manufacturing area. Masters students (8/year, 32 total) complete 2-year projects on industrially-driven project topics, rotating to internships in summer semester to work on scoping and implementation at project partners. Students complete academic training in embedded systems, process modeling, data science, and cloud-based systems design. Their projects are targeted toward sensor retrofit, process monitoring, root cause analysis, and sensor fusion.

Advanced Manufacturing↗

Test Data Monitor

The National Security Campus (NSC) collects a large amount of test data used The National Security Campus (NSC) collects a large amount of test data used to accept high value and high rigor product. The data has been used historically to support root cause analysis when anomalies are detected in down-stream processes. The opportunity to use the data for predictive failure analysis however, had never been exploited. The primary goal of the Test Data Monitor (TDM) software is to provide automated capabilities to analyze data in near-real-time and report trends that foreshadow actual product failures. To date, the aerospace industry as a whole is challenged at utilizing collected data to the degree that modern technology allows. As a result of the innovation behind TDM, Honeywell is able to monitor millions of data points through a multitude of SPC algorithms continuously and autonomously so that our personnel resources can more efficiently and accurately direct their attention to suspect processes or features. TDM’s capabilities have been recognized by our U.S. Department of Energy National Nuclear Security Administration (NNSA) sponsor for potential use at other sites within the NNSA. This activity supports multiple initiatives including expectations of the NNSA and broader corporate goals that center around data-based quality controls on production.

Bosas, Joseph↗

Stack Trace Ahalysis Tool

STAT is a light weight debugging tool that gathers and merges stack traces from all of the processes in a parallel application. STAT uses the MRNet tree based overlay network to broadcast commands from the tool front-end to the STAT daemons and for the front-end to gather the traces from the STAT daemons. As the traces propagate through the MRNet network tree, they are merged across all tasks to form a similar function call patterns and to delineate a small set of equivalence classes. A representative task from each of these classes can then be fed into a full feature debugger like TolalView for root cause analysis.

Lee, Gregory L.↗

Bifacial Photovoltaic Module Degradation Dynamics

In a comprehensive study conducted at NREL's 75 kW bifacial single-axis-tracked field, accelerated degradation was observed in four out of five bifacial silicon photovoltaic (PV) module technologies when compared to their monofacial counterparts. Root cause analysis of accelerated bifacial degradation involved various analytical tools and techniques. This included employing RdTools to identify rates of power loss, conducting measurements on fielded and control modules using infrared imaging, electroluminescence (EL) and photoluminescence (PL), quantum efficiency (QE) analysis, IV-curves assessment, as well as utilizing handheld Raman and reflectance measurement targeted at anti-reflective coating. Most cases pointed to carrier lifetime degradation causing Voc loss and simultaneous Isc loss. In some cases, Isc further decreases likely due to optical effects from encapsulant degradation. The outcomes and methodologies employed in this investigation are documented in this publication. This study's significance is further emphasized by placing the findings within the broader context of the performance and degradation of various bifacial systems, as identified in the PV Fleets data.

bifacial↗

Correlating Log Messages for System Diagnostics

In large-scale computing systems, the sheer volume of log data generated presents daunting challenges for debugging and monitoring of these systems. The Oak Ridge Leadership Computing Facility s premier simulation platform, the Cray XT5 known as Jaguar, can generate a few hundred thousand log entries in less than a minute for many system level events. Determining the root cause of such system events requires analyzing and interpretation of a large number of log messages. Most often, the log messages are best understood when they are interpreted collectively rather than individually. In this paper, we present our approach to interpreting log messages by identifying their commonalities and grouping them into clusters. Given a set of log messages within a time interval, we group the messages based on source, target, and/or error type, and correlate the messages with hardware and application information. We monitor the Lustre log messages in the XT5 console log and show that such grouping of log messages assists in detecting the source of system events. By intelligent grouping and correlation of events in the log, we are able to provide system administrators with meaningful information in a concise format for root cause analysis.

97 MATHEMATICS AND COMPUTING↗

3-Dimensional Root Cause Diagnosis via Co-analysis

With the growth of system size and complexity, reliability has become a major concern for large-scale systems. Upon the occurrence of failure, system administrators typically trace the events in Reliability, Availability, and Serviceability (RAS) logs for root cause diagnosis. However, RAS log only contains limited diagnosis information. Moreover, the manual processing is time-consuming, error-prone, and not scalable. To address the problem, in this paper we present an automated root cause diagnosis mechanism for large-scale HPC systems. Our mechanism examines multiple logs to provide a 3-D fine-grained root cause analysis. Here, 3-D means that our analysis will pinpoint the failure layer, the time, and the location of the event that causes the problem. We evaluate our mechanism by means of real logs collected from a production IBM Blue Gene/P system at Oak Ridge National Laboratory. It successfully identifies failure layer information for 219 failures during 23-month period. Furthermore, it effectively identifies the triggering events with time and location information, even when the triggering events occur hundreds of hours before the resulting failures.

Zheng, Ziming↗

3-Dimensional root cause diagnosis via co-analysis. In: ICAC '12 Proceedings of the 9th international conference on Autonomic computing

With the growth of system size and complexity, reliability has become a major concern for large-scale systems. Upon the occurrence of failure, system administrators typically trace the events in Reliability, Availability, and Serviceability (RAS) logs for root cause diagnosis. However, RAS log only contains limited diagnosis information. Moreover, the manual processing is time-consuming, error-prone, and not scalable. To address the problem, in this paper we present an automated root cause diagnosis mechanism for large-scale HPC systems. Our mechanism examines multiple logs to provide a 3-D fine-grained root cause analysis. Here, 3-D means that our analysis will pinpoint the failure layer, the time, and the location of the event that causes the problem.

Zheng, Ziming↗

Facilitating Data Collection of Maintenance Events to Populate the Hydrogen Component Reliability Database (HyCReD)

The Hydrogen Component Reliability Database (HyCReD) is a collaborative project between the National Renewable Energy Laboratory, the University of Maryland, and hydrogen stakeholders to improve safety and reliability for hydrogen facilities by implementing component reliability data taxonomies that support hydrogen infrastructure failure rate analysis. The project aims to quantify failure rates of hydrogen components through high-quality data collection and analysis on root causes and maintenance needed. HyCReD provides a common database for cataloging hydrogen component failures which exists for reliability research in many other mature industries [2]. The database fills a gap for the hydrogen community by providing a scientifically rigorous approach to quantitative risk assessment (QRA), prognostic health management (PHM), and reliability-centered maintenance (RCM) analysis. High level results will be aggregated and anonymized to protect company sensitive information; detailed results will be used to help address issues of hydrogen components. These advanced analytics will support accelerated deployment of hydrogen infrastructure by enabling better: design and safety of projects (safety codes and standards development), infrastructure reliability and cost (component failure rates, maintenance protocols), and component R&D needs (robust supply chain). A key to a successful HyCReD implementation is facilitating the ease of reporting and data quality in the database that can be used for analysis. Maintenance data was a previously identified gap in initial efforts to populate and validate the database taxonomies [3]. Collection of maintenance data will be instrumental in identifying failure modes and rates, identifying incipient component failures or reduced performance, cataloging best practices for maintenance routines and methods for prognostic health management, and quantifying the risk and effect of different failure modes. Several key priorities are identified for streamlined data collection to achieve quality and detailed failure data: Applicability, Ease of Use, Accessibility, and Information Security. The HyCReD team has now begun deployment of the database to several companies and groups that have signed non-disclosure agreements to facilitate the data collection of failures in industry hydrogen refueling station infrastructure. This paper will provide an update into the process of HyCReD deployment including the development of a coding guide for facility personnel to reference and ensure data quality and consistency from one station to another as well as implementation of contextually dependent data fields of system taxonomy and formatted entries to provide ease of use. The goal is to communicate the lessons learned from the roll-out to technicians and engineers in the field, and the addition of need for high level of security to protect all stakeholders.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Anomaly Detection and Identification Using a Leave-One-Variable-Out Method

At nuclear power plants (NPPs), anomaly detection and identification (i.e., determining the causes of anomalies) are important tasks for ensuring the safe and efficient operation of NPPs. These tasks are currently labor-intensive and costly, and are made more difficult by the size and complexity of NPP systems. An alternative approach to conducting these tasks is to automate them, such as via the reconstruction-based contribution method, which is a well-researched unsupervised machine learning method that uses a data-driven model of anomaly-free behavior to detect events and then identify each variable’s contributions to those events. The present effort developed a novel contribution approach that utilized a leave-one-variable-out (LOVO) model, with which each variable is predicted using all the other variables. The novelty lay in transforming this model into a reconstruction model and modifying the identification algorithm to work with the new reconstruction model. To evaluate this method in a controlled environment, a synthetic dataset based on spring-mass-damper (SMD) systems (commonly found in mechanical engineering references) was used, with known anomalies introduced into the system. The proposed method successfully detected the anomalies and afforded insights into their causes, thus enabling the appropriate identifications to be made.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Unsupervised Process Anomaly Detection and Identification Using the Leave-One-Variable-Out Approach

Automated anomaly detection and identification can signal equipment issues and pinpoint causes in large-scale industrial systems. For systems with limited failure history, unsupervised machine learning methods can be utilized as they do not require past failures. This study introduces the leave-one-variable-out (LOVO) model, which masks one variable at a time to predict the others, learning underlying process correlations. Detection performance was assessed with synthetic and experimental data, while identification performance used only synthetic data due to its ability to generate labeled anomaly types. For detection using synthetic data, the LOVO model generally outperformed comparative models; while using experimental data, the comparative methods outperformed the LOVO model. However, the comparative methods required selecting a latent size, and these conclusions pertain to using the optimal size. In practice, it would not be feasible to always select the optimal value, and incorrect selections impacted performance. In contrast, the LOVO model does not require a latent space. For identification using synthetic data, the LOVO model was slightly outperformed in interpretability and repeatability but still demonstrated impressive results. These outcomes suggest that the LOVO model is an effective model and may be more easily implemented without the challenging tuning process of selecting a latent size.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Using Semantic Web Technologies to Develop Intrinsically Resilient Energy Control Systems

To preserve critical energy control functions while under attack, it is necessary to perform comprehensive analysis on root causes and impacts of cyber intrusions without sacrificing the availability of energy delivery. We propose to design an intrinsically resilient energy control system where we extensively utilize Semantic Web technologies, which play critical roles in knowledge representation and acquisition. While our ultimate goal is to ensure availability/resiliency of energy delivery functions and the capability to assess root causes and impacts of cyber intrusions, the focus of this paper is to demonstrate a proof of concept of how Semantic Web technologies can significantly contribute to resilient energy control systems.

cybersecurity↗

A Zoom-in Analysis of I/O Logs to Detect Root Causes of I/O Performance Bottlenecks

Scientific applications frequently spend a large fraction of their execution time in reading and writing data on parallel file systems. Identifying these I/O performance bottlenecks and attributing root causes are critical steps toward devising optimization strategies. Several existing studies analyze I/O logs of a set of benchmarks or applications that were run with controlled behaviors. However, there is still a lack of general approach that systematically identifies I/O performance bottlenecks for applications running "in the wild" on production systems. In this study, we have developed an analysis approach of "zooming in" from platform-wide to application-wide to job-level I/O logs for identifying I/O bottlenecks in arbitrary scientific applications. We analyze the logs collected on a Cray XC40 system in production over a two-month period. This study results in several insights for application developers to use in optimizing I/O behavior.

Wang, Teng↗

Investigation and Recovery Following the Drum Event at INL's ARP-V Facility - 19517

Contractors working under the Department of Energy's (DoE's) Idaho Field Office have been exhuming and processing buried legacy waste from early weapons development and productions operations since 1984. Since operations began, nearly 400,000 drums have passed through this process. On April 11, 2018, repackaged drums were staged in an airlock (radiological buffer area) awaiting transfer to another facility for characterization. Late in the evening after operations had ended, four of the drums over pressurized causing an energetic displacement of their lids. A significant fraction of drum contents was ejected and dispersed throughout the airlock. The event activated the fire detection system in the facility initiating an emergency response from the site fire department, radiological protection, and operations. Initial intervention by fire department personnel was unsuccessful at quenching the reaction, so the responders egressed to a safe distance and monitored the situation. After determining the event had terminated and it was safe to enter, the facility was assessed revealing significant contamination in the airlock from the ejected waste. The area surrounding the facility was surveyed and it was determined that no contamination was released outside the facility. The Fluor Idaho, LLC (Fluor Idaho) Program Manager commissioned an investigation and recovery effort, which focused on three aspects: (1) technical investigation, (2) event and causal analysis, and (3) facility recovery. A team of highly qualified individuals from around the DOE Complex was assembled to support the technical investigation. The team included representatives from several Fluor Idaho, Fluor Corporate, Battelle Energy Alliance, LLC (INL), LANL, SRNL, South West Research Institute, DOE Carlsbad Field Office, and Defense Nuclear Facility Safety Board, and DOE-Headquarters. This team addressed the reaction mechanism of the event by guiding the sampling activities and coordinating analysis and testing by the analytical laboratories. Independent and experienced investigators were brought in to conduct a rigorous event and cause analysis. The Causal Analysis Team pursued direct, root, and contributing causes. Interviews were conducted, and documents were reviewed to develop an event and causal analysis. The Accelerated Retrieval Project V Operations team mobilized to decontaminate and recover the facility. Significant progress has been made in this effort, area and airborne contamination levels have been significantly reduced allowing limited activities to resume. In addition, this group supported sample collection efforts as directed by the Technical Investigation Team. This paper reviews key findings and milestones from the technical investigation, the event and causal analysis, and facility recovery operations. From these elements of the investigation, lessons learned are identified that would benefit other organizations involved with legacy waste management and disposition. Waste management is common to all sites in the DOE Complex and the events and recovery efforts described in this paper may provide beneficial insights to these operations. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Understanding Peelle’s Pertinent Puzzle bias in generalized least squares regression through eigenspectrum analysis

Certain correlation structures in the data covariance matrix (DCM) used for generalized least squares (GLS) regression can result in biased estimates, commonly known in the field of nuclear data evaluation as Peele’s Pertinent Puzzle (PPP). This article introduces a generative, forward modeling framework within which the PPP bias is characterized through an eigenspectrum analysis of the DCM. This analysis highlights the root cause of the bias, generalizes the problem beyond the nuclear data field, and provides insight to the problem regimes where it can occur. What follows is an understanding that the bias can show up for any experimental neutron time-of-flight data for which systematic uncertainties have been quantified. Lastly, a discussion of the adaptation of cross validation approaches that require pre-whitening to incorporate the known ‘fix’ to the PPP bias in the GLS estimator.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Study of the Heater-Coil Electrical Insulation for the HL-LHC Low Beta Quadrupoles

In the framework of the HL-LHC project, the present LHC low-β superconducting quadrupoles will be substituted with higher performance Nb 3 Sn magnets (MQXF) with 11.4 T coil peak field. MQXF coils are impregnated with epoxy resin to reduce risk of stress concentration on the brittle conductor. The magnet quench protection is provided by CLIQ and quench heaters to ensure a redundant system. Quench heaters are impregnated with the coils in order to have suitable thermal contact with them, and to prevent the hot spot temperature from exceeding 350K during normal operation in case of a quench. Quench heaters are insulated from the coil by S-2 Glass and polyimide. Here, the test of the first MQXF prototype (4 m long) MQXFAP1 was stopped by a coil-to-ground short circuit triggered by a heater-to-coil short. This issue triggered a root analysis of the causes of this short. Here we prove that the use of a non-conforming cloth in coil impregnation, further weakened by non-conforming high voltage test, has triggered the shorts. Moreover, we present an analysis of the heater-to-coil insulation strength, showing the role of blistering phenomena and how they are triggered by a combination of magnet powering and heater firing.

43 PARTICLE ACCELERATORS↗

Ground observations of a space laser for the assessment of its in-orbit performance

The wind mission Aeolus of the European Space Agency was a groundbreaking achievement for Earth observation. Between 2018 and 2023, the space-borne lidar instrument ALADIN onboard the Aeolus satellite measured atmospheric wind profiles with global coverage, which contributed to improving the accuracy of numerical weather prediction. The precision of the wind observations, however, declined over the course of the mission due to a progressive loss of the atmospheric backscatter signal. The analysis of the root cause was supported by the Pierre Auger Observatory in Argentina whose fluorescence detector registered the ultraviolet laser pulses emitted from the instrument in space, thereby offering an estimation of the laser energy at the exit of the instrument for several days in 2019, 2020, and 2021. The reconstruction of the laser beam not only allowed for an independent assessment of the Aeolus performance, but also helped to improve the accuracy in the determination of the laser beam’s ground track on single pulse level. The results presented in this paper set a precedent for the monitoring of space lasers by ground-based telescopes and open new possibilities for the calibration of cosmic-ray observatories.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗