Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “software failure”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Dynamic Contingency Analysis Tool 2.0 User Manual with Test System Examples

This document introduces Version 2 of the Dynamic Contingency Analysis Tool (DCAT) software package developed by Pacific Northwest National Laboratory and serves as a guide for using this package to conduct cascading failure simulations for the electric power system. The DCAT is an open-platform and publicly available software; it is intended to help develop applications that aim to improve the capabilities of power system planning engineers to assess the impact and likelihood of extreme contingencies and potential cascading events across their systems and interconnections. Outputs from the DCAT will support finding mitigating actions to reduce the risk of cascading outages in technically sound and effective ways. This manual provides detailed instructions on how to use this tool on multicore Windows workstations or servers. In the public version of DCAT, several improvements have been made over the previous version: (1) it was modularized; (2) a graphical user interface (GUI) was developed; (3) a new version of the MPjobs parallel processing module was incorporated; and (4) a batch-processing function was added. The features of the DCAT GUI are described, including some examples of how to use the GUI to perform simulations. A full explanation of the DCAT methodology is available in the Phase 1 report.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Fault propagation and effects analysis for designing an online monitoring system for the secondary loop of a nuclear power plant part of a hybrid energy system - 393

This paper studies the propagation and effects of faults of critical components that pertain to the secondary loop of a nuclear power plant found in Nuclear Hybrid Energy Systems (NHES). This information is used to design an on-line monitoring (OLM) system which is capable of detecting and forecasting faults that are likely to occur during NHES operation. In this research, the causes, features, and effects of possible faults are investigated by simulating the propagation of faults in the secondary loop. The simulation is accomplished by using the Integrated System Failure Analysis (ISFA). ISFA is used for analyzing hardware and software faults during the conceptual design phase. In this paper, the models of system components required by ISFA are initially constructed. Then, the fault propagation analysis is implemented, which is conducted under the bounds set by acceptance criteria derived from the design of an OLM system. The result of the fault simulation is utilized to build a database for fault detection and diagnosis, provide preventive measures, and propose an optimization plan for the OLM system. (authors)

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

A Report on Simulation-Driven Reliability and Failure Analysis of Large-Scale Storage Systems

High-performance computing (HPC) storage systems provide data availability and reliability using various hardware and software fault tolerance techniques. Usually, reliability and availability are calculated at the subsystem or component level using limited metrics such as, mean time to failure (MTTF) or mean time to data loss (MTTDL). This often means settling on simple and disconnected failure models (such as exponential failure rate) to achieve tractable and close-formed solutions. However, such models have been shown to be insufficient in assessing end-to-end storage system reliability and availability. We propose a generic simulation framework aimed at analyzing the reliability and availability of storage systems at scale, and investigating what-if scenarios. The framework is designed for an end-to-end storage system, accommodating the various components and subsystems, their interconnections, failure patterns and propagation, and performs dependency analysis to capture a wide-range of failure cases. We evaluate the framework against a large-scale storage system that is in production and analyze its failure projections toward and beyond the end of lifecycle. We also examine the potential operational impact by studying how different types of components affect the overall system reliability and availability, and present the preliminary results

97 MATHEMATICS AND COMPUTING↗

Characterization of Orography-Influenced Riming and Secondary Ice Production and Their Effects on Precipitation Rates Using Radar Polarimetry and Doppler Spectra (CORSIPP-SAIL)

The Characterization of Orography-Influenced Riming and Secondary Ice Production and Their Effects on Precipitation Rates Using Radar Polarimetry and Doppler Spectra (CORSIPP) project was conducted to help improve the understanding of precipitation formation in orographically influenced terrain. Special focus is put on the two processes of riming and secondary ice production and their external drivers. Two instruments, a polarimetric W-band simultaneous transmission simultaneous reception (STSR) Doppler cloud radar manufactured by Radiometer Physics GmbH (RPG, instrument type RPG-FMCW-94-DP), from now on named LIMRAD94, and the video in situ snowfall sensor (VISSS), were deployed at the U. S. Department of Energy Atmospheric Radiation Measurement (ARM) user facility’s Surface Atmosphere Integrated Field Laboratory (SAIL) site in Gothic, Colorado between November 2022 and June 2023 during the second SAIL winter. Note that the exact dates of data availability differ between the instruments. Both instruments arrived at Gothic on November 2, 2022, 09:20 local time. VISSS, described by Maahn et al., is equipped with two camera systems with telecentric lenses. The two cameras are at a 90° angle to each other. This configuration allows for size-independent measurements by capturing images of hydrometeors from two sides at a high frame rate of 250 Hz. With a minimum detection size of 200 μm, VISSS provides valuable insights into particle size, number, shape, complexity, and fall velocity. The VISSS was deployed on the grassland next to the ARM facility with the amazing help of the ARM employees on site. The setup started on November 2, 2022, and was finished on November 5, 2022, without major problems. VISSS measurements were started on November 6, 2022. LIMRAD94 was installed on a scaffolding platform near Orehouse (Gothic) on November 9, 2022, with the great help of RMBL staff. LIMRAD94 was mounted on a cold temperature scanner prototype. After a short test of the setup on November 9, 2022, the digital control of the scanner elevation stopped working for (at that time) unknown reasons. All attempts to resolve the problem failed. This malfunction made it impossible to operate LIMRAD94 in scanning mode. The scanner was then manually moved to zenith pointing mode and between November 10 and November 15, 2022, vertical observations for polarimetric calibration were performed. On November 15, 2022, after the polarimetric calibration was applied, the scanner was manually moved to 40° elevation with azimuthal view towards the Ka-band ARM Zenith Radar (KAZR) and measurements were continued at constant elevation. Investigation of the scanner malfunction on February 6, 2023, by Benn Schmatz revealed a disengagement between the cogwheel of the elevation motor and the cogwheel moving the scanner in elevation. This mechanical issue was temporarily solved by re-engaging the cogwheels. This made the scanner operational again for about four weeks, until mechanical force disengaged the cogwheels again on March 15, 2023. This repeated scanner failure remained undetected for about three weeks until April 8, 2023; during this time the scanner was stuck at 72° elevation. However, the radar software continued to produce data files falsely indicating that the scanner was still operational. After the scanner failure was noticed, scanning was stopped again, and we returned to constant elevation measurements. On May 15, 2023, the cogwheels of the elevation motor were secured with additional screws sent by the manufacturer. At some point in May, the cogwheels of the azimuth motor were also disengaged by mechanical force, which still allowed for range height indicator (RHI) but no plan position indicator (PPI) scans in the last weeks of the campaign. The azimuth motor was repaired in Germany after the end of the campaign. Throughout the campaign, RMBL and ARM staff kept the radar and the VISSS free of snow.

54 ENVIRONMENTAL SCIENCES↗

Operational experience with the new CMS DAQ-Expert

The data acquisition (DAQ) system of the Compact Muon Solenoid (CMS) at CERN reads out the detector at the level-1 trigger accept rate of 100 kHz, assembles events with a bandwidth of 200 GB/s, provides these events to the high level-trigger running on a farm of about 30k cores and records the accepted events. Comprising custom-built and cutting edge commercial hardware and several 1000 instances of software applications, the DAQ system is complex in itself and failures cannot be completely excluded. Moreover, problems in the readout of the detectors,in the first level trigger system or in the high level trigger may provoke anomalous behaviour of the DAQ systemwhich sometimes cannot easily be differentiated from a problem in the DAQ system itself. In order to achieve high data taking efficiency with operators from the entire collaboration and without relying too heavily on the on-call experts, an expert system, the DAQ-Expert, has been developed that can pinpoint the source of most failures and give advice to the shift crew on how to recover in the quickest way. The DAQ-Expert constantly analyzes monitoring data from the DAQ system and the high level trigger by making use of logic modules written in Java that encapsulate the expert knowledge about potential operational problems. The results of the reasoning are presented to the operator in a web-based dashboard, may trigger sound alerts in the control room and are archived for post-mortem analysis - presented in a web-based timeline browser. We present the design of the DAQ-Expert and report on the operational experience since 2017, when it was first put into production.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Exploratory Nuclear Reactor Safety Analysis and Visualization via Integrated Topological and Geometric Techniques

A recent trend in the nuclear power engineering field is the implementation of heavily computational and time consuming algorithms and codes for both design and safety analysis. In particular, the new generation of system analysis codes aim to embrace several phenomena such as thermo-hydraulic, structural behavior, and system dynamics, as well as uncertainty quantification and sensitivity analyses. The use of dynamic probabilistic risk assessment (PRA) methodologies allows a systematic approach to uncertainty quantification. Dynamic methodologies in PRA account for possible coupling between triggered or stochastic events through explicit consideration of the time element in system evolution, often through the use of dynamic system models (simulators). They are usually needed when the system has more than one failure mode, control loops, and/or hardware/process/software/human interaction. Dynamic methodologies are also capable of modeling the consequences of epistemic and aleatory uncertainties. The Monte-Carlo (MC) and the Dynamic Event Tree (DET) approaches belong to this new class of dynamic PRA methodologies. The major challenges in using MC and DET methodologies (as well as other dynamic methodologies) are the heavier computational and memory requirements compared to the classical ET analysis. This is due to the fact that each branch generated can contain time evolutions of a large number of variables (about 50,000 data channels are typically present in RELAP) and a large number of scenarios can be generated from a single initiating event (possibly on the order of hundreds or even thousands). Such large amounts of information are usually very difficult to organize in order to identify the main trends in scenario evolutions and the main risk contributors for each initiating event. This report aims to improve Dynamic PRA methodologies by tackling the two challenges mentioned above using: 1) adaptive sampling techniques to reduce computational cost of the analysis and 2) topology-based methodologies to interactively visualize multidimensional data and extract risk-informed insights. Regarding item 1) we employ learning algorithms that aim to infer/predict simulation outcome and decide the coordinate in the input space of the next sample that maximize the amount of information that can be gained from it. Such methodologies can be used to both explore and exploit the input space. The later one is especially used for safety analysis scopes to focus samples along the limit surface, i.e. the boundaries in the input space between system failure and system success. Regarding item 2) we present a software tool that is designed to analyze multi-dimensional data. We model a large-scale nuclear simulation dataset as a high-dimensional scalar function defined over a discrete sample of the domain. First, we provide structural analysis of such a function at multiple scales and provide insight into the relationship between the input parameters and the output. Second, we enable exploratory analysis for users, where we help the users to differentiate features from noise through multi-scale analysis on an interactive platform, based on domain knowledge and data characterization. Our analysis is performed by exploiting the topological and geometric properties of the domain, building statistical models based on its topological segmentations and providing interactive visual interfaces to facilitate such explorations.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

FAULT PROPAGATION AND EFFECTS ANALYSIS FOR DESIGNING AN ONLINE MONITORING SYSTEM FOR THE SECONDARY LOOP OF A NUCLEAR POWER PLANT PART OF A HYBRID ENERGY SYSTEM

This paper studies the propagation and effects of faults of critical components that pertain to the secondary loop of a nuclear power plant found in Nuclear Hybrid Energy Systems (NHES). This information is used to design an on-line monitoring (OLM) system which is capable of detecting and forecasting faults that are likely to occur during NHES operation. In this research, the causes, features, and effects of possible faults are investigated by simulating the propagation of faults in the secondary loop. The simulation is accomplished by using the Integrated System Failure Analysis (ISFA). ISFA is used for analyzing hardware and software faults during the conceptual design phase. In this paper, the models of system components required by ISFA are initially constructed. Then, the fault propagation analysis is implemented, which is conducted under the bounds set by acceptance criteria derived from the design of an OLM system. The result of the fault simulation is utilized to build a database for fault detection and diagnosis, provide preventive measures, and propose an optimization plan for the OLM system.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

URBAN-NET-TOOLKIT

Strengthening and maintaining secure, functioning, and resilient Critical Infrastructure Systems (CISs) is one of our nation's primary goals, and it requires proactive and coo0rdinated efforts among federal, state, local, public and private owners and operators of CISs. Recent worldwide extreme events have shown how interdependencies across CISs (electric grid, telecommunications, transportation, water networks, etc.) can drastically increase system vulnerability. To understand these complex interdependencies, ORNL has instantiated Urban-NET, a novel decision support system for CIS operators who need to identify vulnerabilities for minimizing the cascading impacts - saving utilities, the federal and state government, and private sector companies potentially billions by avoiding a complete CI failure and/or shutdown. The developed system includes a stack of software tools for preprocessing, modeling, interconnecting multiple CIS datasets as an integrated heterogeneous graph (i.e. network), identifying vulnerabilities through topological analysis, and predicting consequences based on simulations along with visualization.

Chinthavali, Supriya↗

Seismic monitoring of EGS tests at the Coso Geothermal area, California, using accurate MEQ locations and full moment tensors

In this study, we studied high-resolution relative locations and full moment tensors of microearthquakes (MEQs) occurring before, during and following Enhanced Geothermal Systems (EGS) experiments in two wells at the Coso geothermal area, California. The objective was to map new fractures, determine the mode and sense of failure, and characterize the stress cycle associated with injection. New software developed for this work combines waveform cross-correlation measurement of arrival times with relative relocation methods, and assesses confidence regions for moment tensors derived using linear-programming methods. For moment tensor determination we also developed a convenient Graphical User Interface (GUI), to streamline the work. We used data from the U.S. Navy’s permanent network of three-component digital borehole seismometers and from 14 portable three-component digital instruments. The latter supplemented the permanent network during injection experiments in well 34A-9 in 2004 and well 34-9RD2 in 2005. In the experiment in well 34A-9, the co-injection earthquakes were more numerous, smaller, more explosive and had more horizontal motion, compared with the pre-injection earthquakes. In the experiment in well 34-9RD2 the relocated hypocenters reveal a well-defined planar structure, 700 m long and 600 m high in the depth range 0.8 to 1.4 km below sea level, striking N 20° E and dipping at 75° to the WNW. The moment tensors show that it corresponds to a mode I (opening) crack. Finally, for both wells, the perturbed stress state near the bottom of the well persisted for at least two months following the injection.

15 GEOTHERMAL ENERGY↗

Holistic Measurement Driven Resilience: Combining Operational Fault and Failure Measurements and Fault Injection for Quantifying Fault Detection, Propagation and Impact. Final report

For HPC systems to date, application resilience to faults and failures has been accomplished by the brute- force method of checkpoint/restart, which allows an application to make forward progress in the face of system and application faults, errors, and failures independent of root cause or end result. It has remained the primary resilience mechanism because we lack a way to identify faults and anticipate consequences early enough to take meaningful mitigating action. However, checkpoint/restart implementations put a tremendous burden on system resources and on the applications themselves and is becoming less feasible at scale. Because we have not yet operated at scales at which checkpoint/restart fails to provide forward progress, despite increasing costs, vendors have had little motivation to provide the instrumentation necessary for early identification of faults and failures. However, as we move from petascale to exascale, component mean time to failure (MTTF) will render the existing techniques ineffectual and/or too expensive. Furthermore, fault recovery mechanisms such as failover and/or error correction introduce performance inconsistency. Instrumentation allowing early indication of problems and tools to enable use of such information by systems, operating systems, and applications offer an alternative, more scalable and less costly solution. In the HMDR project, we built on our experience and expertise developed and accumulated over years of research on design, monitoring, measurement, and assessment of resilient computing systems. Analysis of field data on the current and past generations of extreme-scale systems revealed several challenges that, if not addressed in increasingly larger and more complex systems, may hinder the effectiveness of future exascale computing systems. Specifically, i) file systems and interconnects in current-generation large-scale systems already operate at the margins of resiliency, including consistent performance, and may not scale to larger deployments; ii) automated, software-based failover mechanisms are frequently inadequate and can introduce wider failures, such that failures during recovery may lead to system/application failures, including system-wide outages; and iii) silent data corruption represents a critical fault mode and will require efficient detection mechanisms if next-generation applications are to take full advantage of exascale hardware. To address the above challenges, we assembled a team of world-renowned experts in resilient extreme- scale computing from the University of Illinois (Electrical and Computer Engineering, Computer Science, and NCSA), SNL, LANL, NERSC, and Cray. Our team includes representatives from centers that house many of the largest HPC resources in the world, both today and over the coming years. The team has a unique track record of research in i) system and application failure characterization based on the analysis of field data, ii) data-driven design of fault/error detection mechanisms, and iii) experimental characterization of system/application resiliency. The team includes system owners/operators who provide continuous data collection and access and ensure installation of appropriate analysis tools.

97 MATHEMATICS AND COMPUTING↗

Probabilistic Assessment of Structural Integrity

A probability-based approach, combining deterministic and probabilistic methods, was developed for analyzing building and component failures, which are especially crucial for complex structures like nuclear power plants. This method links finite element and probabilistic software to assess structural integrity under static and dynamic loads. This study uses NEPTUNE software, which is validated, for a deterministic transient analysis and ProFES software for probabilistic models. In a case study, deterministic analyses with varied random variables were transferred to ProFES for probabilistic analyses of piping failure and wall damage. A Monte Carlo Simulation, First-Order Reliability Method, and combined methods were employed for probabilistic analyses under severe transient loading, focusing on a postulated accident at the Ignalina Nuclear Power Plant. The study considered uncertainties in material properties, component geometry, and loads. The results showed the Monte Carlo Simulation method to be conservative for high failure probabilities but less so for low probabilities. The Response Surface/Monte Carlo Simulation method explored the impact load–failure probability relationship. Given the uncertainties in material properties and loads in complex structures, a deterministic analysis alone is insufficient. Probabilistic analysis is imperative for extreme loading events and credible structural safety evaluations.

Mathematics↗

The False Dilemma: Rethinking AF Science and Technology Officer Talent Management

In 2019, Secretary of the Air Force (SECAF) Heather Wilson launched the 2030 Science and Technology (S&T) Strategy by stating, “The advantage will go to those who create the best technologies and who integrate and field them in creative operational ways that provide military advantages.” In 2021, 15% of Air Force general officers responsible for creating and integrating this technological edge had a science, technology, engineering, and math (STEM) graduate degree, and less than 1% of generals had a STEM doctorate. As one point of comparison, at least 32% and 13% of founding CEOs of Fortune 500 technology companies had a master’s and doctoral STEM degree, respectively. The gulf is larger when you consider that these CEOs used their technical degrees throughout their careers whereas most general officers do not. In large organizations, if the leader does not possess technical knowledge, it is difficult to drive innovation systems and connect ideas to reality. The evidence for this disconnect is abundant in acquisition challenges for high-tech systems, significant pushback to the “revolution in military affairs”, and failure of the “Third Offset” to take hold and deliver capabilities to offset Chinese and Russian capabilities. Indeed, when the former Air Force and Space Force Chief Software Officer, Nicolas M. Chaillan, offered his resignation, one of the main reasons for leaving was “the failure of OSD and the Joint Staff to deliver on their own alleged top ‘priority’, JADC2 – they couldn’t ‘walk the walk.’” Senior leaders have recognized this disconnect, and several inquiries and studies regarding STEM competency have been conducted. Many have posed a false dilemma: a large, STEM-cognizant or small, STEM-expert force. However, current and future STEM human capital needs are managed at the unit and functional area level. This leads to “silos” and the tactical “needs of the present” dominating the strategic “needs of the future”, highlighted by only two functional areas systematically tracking future STEM needs. Therefore, the decision on force structure appears to have been historically made by default through these and other perceived structural realities, and a vast majority of recommendations have focused on the small STEM community – an exercise in diminishing returns. In essence, the solution often boils down to finding unicorns – officers that have chartered the unforgiving pathway through the traditional “gates” to general officer while obtaining sufficient STEM proficiency along the way. Fortunately for the Air Force, these officers exist, albeit at a rate below what the evidence would suggest is necessary. However, this approach fundamentally limits the ability to develop a deep pool of officers with the ability to create and integrate technologies and ensures the Dunning-Kruger effect is prevalent. The Dunning-Kruger effect is a cognitive bias where those with limited knowledge in an area lack the expertise necessary to recognize their lack of expertise and consequently are prone to overestimate their knowledge and performance. In short, we need less leaders overestimating their STEM knowledge and more with hard-earned STEM competency required to “walk the walk” to create and integrate technologies for military advantage. This will not be accomplished by restricting the emphasis to the traditional scientists and engineering (S&E) career fields – the pool, only 10% of line officers, and general officer progression is just too small. Fortunately, the recent SECAF’s Management Initiatives and Chief of Staff’s (CSAF) Action Orders, both containing emphasis on organic expertise to “accrue advantage in military-technological competition,” present an opportunity to truly develop a framework for the force of the future.

99 GENERAL AND MISCELLANEOUS↗

Field Experience Detecting PV Underperformance in Real Time Using Existing Instrumentation

Maintenance at large-scale photovoltaic plants employs a mix of preventative and corrective maintenance practices. Large outages, such as an inverter tripping offline, are often easy to detect. More subtle sub-inverter faults and failures can accumulate and go unnoticed for months or years. A software-based fault detection method has been developed to analyze commonly measured data from large-scale PV plants for more timely detection of subtle underperformance. The method has been demonstrated on eight datasets from large-scale plants with high accuracy of detection. Results are validated using aerial infrared scanning. String outages are detected with a true positive rate of 73 percent and tracker issues are detected with a true positive rate of 88 percent. The developed method can be uniformly applied to photovoltaic plants across a range of scales and configurations to assess performance, quickly detect underperformance, and determine the source and location of failures. The results inform and improve operations and maintenance at PV plants, ultimately aiding in improved affordability, reliability, availability, and resiliency of solar electricity.

14 SOLAR ENERGY↗

RAVEN, a New Software for Dynamic Risk Analysis

RAVEN is a generic software driver to perform parametric and probabilistic analysis of code simulating complex systems. Initially developed to provide dynamic risk analysis capabilities to the RELAP-7 code [1] is currently being generalized with the addition of Application Programming Interfaces (APIs). These interfaces are used to extend RAVEN capabilities to any software as long as all the parameters that need to be perturbed are accessible by inputs files or directly via python interfaces. RAVEN is capable to investigate the system response probing the input space using Monte Carlo, grid strategies, or Latin Hyper Cube schemes, but its strength is its focus toward system feature discovery like limit surfaces separating regions of the input space leading to system failure using dynamic supervised learning techniques. The paper will present an overview of the software capabilities and their implementation schemes followed by same application examples.

97 MATHEMATICS AND COMPUTING↗

Final Technical Report on Quantifying Dependability Attributes of Software Based Safety Critical Instrumentation and Control Systems in Nuclear Power Plants

With the current transition from analog to digital instrumentation and control systems in nuclear power plants, the number and variety of software-based systems have significantly increased. The sophisticated nature and increasing complexity of software raises trust in these systems as a significant challenge. The trust placed in a software system is typically termed software dependability. Software dependability analysis faces uncommon challenges since software systems’ characteristics differ from those of hardware systems. The lack of systematic science-based methods for quantifying the dependability attributes in software-based instrumentation as well as control systems in safety critical applications has proved itself to be a significant inhibitor to the expanded use of modern digital technology in the nuclear industry. Dependability refers to the ability of a system to deliver a service that can be trusted. Dependability is commonly considered as a general concept that encompasses different attributes, e.g., reliability, safety, security, availability and maintainability. Dependability research has progressed significantly over the last few decades. For example, various assessment models and/or design approaches have been proposed for software reliability, software availability and software maintainability. Advances have also been made to integrate multiple dependability attributes, e.g., integrating security with other dependability attributes, measuring availability and maintainability, modeling reliability and availability, quantifying reliability and security, exploring the dependencies between security and safety and developing integrated analysis models. However, there is still a lack of understanding of the dependencies between various dependability attributes as a whole and of how such dependencies are formed. To address the need for quantification and give a more objective basis to the review process -- therefore reducing regulatory uncertainty -- measures and methods are needed to assess dependability attributes early on, as well as throughout the life-cycle process of software development. In this research, extensive expert opinion elicitation is used to identify the measures and methods for assessing software dependability. Semi-structured questionnaires were designed to elicit expert knowledge. A new notation system, Causal Mechanism Graphing, was developed to extract and represent such knowledge. The Causal Mechanism Graphs were merged, thus, obtaining the consensus knowledge shared by the domain experts. In this report, we focus on how software contributes to dependability. However, software dependability is not discussed separately from the context of systems or socio-technical systems. Specifically, this report focuses on software dependability, reliability, safety, security, availability, and maintainability. Our research was conducted in the sequence of stages found below. Each stage is further examined in its corresponding chapter. Stage 1 (Chapter 2): Elicitation of causal maps describing the dependencies between dependability attributes. These causal maps were constructed using expert opinion elicitation. This chapter describes the expert opinion elicitation process, the questionnaire design, the causal map construction method and the causal maps obtained. Stage 2 (Chapter 3): Elicitation of the causal map describing the occurrence of the event of interest for each dependability attribute. The causal mechanisms for the “event of interest” were extracted for each of the software dependability attributes. The “event of interest” for a dependability attribute is generally considered to be the “attribute failure”, e.g. security failure. The extraction was based on the analysis of expert elicitation results obtained in Stage 1. Stage 3 (Chapter 4): Identification of relevant measurements. Measures for the “events of interest” and their causal mechanisms were obtained from expert opinion elicitation for each of the software dependability attributes. The measures extracted are presented in this chapter. Stage 4 (Chapter 5): Assessment of the coverage of the causal maps via measures. Coverage was assessed to determine whether the measures obtained were sufficient to quantify software dependability, and what measures are further required. Stage 5 (Chapter 6): Identification of “missing” measures and measurement approaches for concepts not covered. New measures, for concepts that had not been covered sufficiently as determined in Stage 4, were identified using supplementary expert opinion elicitation as well as literature reviews. Stage 6 (Chapter 7): Building of a detailed quantification model based on the causal maps and measurements obtained. Ability to derive such a quantification model shows that the causal models and measurements derived from the previous stages (Stage 1 to Stage 5) can form the technical basis for developing dependability quantification models. Scope restrictions have led us to prioritize this demonstration effort. The demonstration was focused on a critical system, i.e. the reactor protection system. For this system, a ranking of the software dependability attributes by nuclear stakeholders was developed. As expected for this application, the stakeholder ranking identified safety as the most critical attribute to be quantified. A safety quantification model limited to the requirements phase of development was built. Two case studies were conducted for verification. A preliminary control gate for software safety for the requirements stage was proposed and applied to the first case study. The control gate allows a cost effective selection of the duration of the requirements phase.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

FOX: A Fault Oblivious Extreme-Scale Execution Environment

The FOX project, funded under the ASCR X-stack I program, developed systems software and runtime libraries for a new approach to the data and work distribution for massively parallel, fault oblivious application execution. Our work was motivated by the premise that exascale computing systems will provide a thousand-fold increase in parallelism and a proportional increase in failure rate relative to today’s machines. To deliver the capability of exascale hardware, the systems software must provide the infrastructure to support existing applications while simultaneously enabling efficient execution of new programming models that naturally express dynamic, adaptive, irregular computation; coupled simulations; and massive data analysis in a highly unreliable hardware environment with billions of threads of execution. Our OS research has prototyped new methods to provide efficient resource sharing, synchronization, and protection in a many-core compute node. We have experimented with alternative task/dataflow programming models and shown scalability in some cases to hundreds of thousands of cores. Much of our software is in active development through open source projects. Concepts from FOX are being pursued in next generation exascale operating systems. Our OS work focused on adaptive, application tailored OS services optimized for multi → many core processors. We developed a new operating system NIX that supports role-based allocation of cores to processes which was released to open source. We contributed to the IBM FusedOS project, which promoted the concept of latency-optimized and throughput-optimized cores. We built a task queue library based on distributed, fault tolerant key-value store and identified scaling issues. A second fault tolerant task parallel library was developed, based on the Linda tuple space model, that used low level interconnect primitives for optimized communication. We designed fault tolerance mechanisms for task parallel computations employing work stealing for load balancing that scaled to the largest existing supercomputers. Finally, we implemented the Elastic Building Blocks runtime, a library to manage object-oriented distributed software components. To support the research, we won two INCITE awards for time on Intrepid (BG/P) and Mira (BG/Q). Much of our work has had impact in the OS and runtime community through the ASCR Exascale OS/R workshop and report, leading to the research agenda of the Exascale OS/R program. Our project was, however, also affected by attrition of multiple PIs. While the PIs continued to participate and offer guidance as time permitted, losing these key individuals was unfortunate both for the project and for the DOE HPC community.

97 MATHEMATICS AND COMPUTING↗

Online Diagnosis of Performance Variation in HPC Systems Using Machine Learning

As the size and complexity of HPC systems grow in line with advancements in hardware and software technology, HPC systems increasingly suffer from performance variation due to shared resource contention as well as software- and hardware-related problems. Such performance variations can lead to failures and inefficiencies, and are among the main challenges in system resiliency. To minimize the impact of performance variation, one must quickly and accurately detect and diagnose the anomalies that cause the variation and take mitigating actions. However, it is difficult to identify anomalies based on the voluminous, high-dimensional, and noisy data collected by system monitoring infrastructures. This paper presents a novel machine learning based framework to automatically diagnose performance anomalies at runtime. Our framework leverages historical resource usage data to extract signatures of previously-observed anomalies. We first convert the collected time series data into easy-to-compute statistical features. We then identify the features that are required to detect anomalies, and extract the signatures of these anomalies. At runtime, we use these signatures to diagnose anomalies with negligible overhead. Here, we evaluate our framework using experiments on a real-world HPC supercomputer and demonstrate that our approach successfully identifies 98% of injected anomalies and consistently outperforms existing anomaly diagnosis techniques.

97 MATHEMATICS AND COMPUTING↗

A Reliability Study on the ALERTUS Emergency Management Notification System

Sandia National Laboratories conducted a reliability analysis on the Alertus mass notification system to determine if improvements need to be made to the system to increase reliability. The Alertus mass notification system for Building 803 was analyzed with a set number of components. The components, their associated failure modes and failure mode rates were inputted into a fault tree in the SAPHIRE software which calculated the reliability of the system to be 0.998269.

97 MATHEMATICS AND COMPUTING↗