Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “FAILURES”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

Bladder Cancer Patterns of Pelvic Failure: Implications for Adjuvant Radiation Therapy

Purpose: Local-regional failures (LFs) after cystectomy with or without chemotherapy are common in locally advanced disease. Adjuvant radiation therapy (RT) could reduce LFs, but toxicity has discouraged its use. Modern RT techniques with improved normal tissue sparing have rekindled interest but require knowledge of pelvic failure patterns to design treatment volumes. Methods and Materials: Five-year LF rates after radical cystectomy plus pelvic node dissection with or without chemotherapy were determined for 8 pelvic sites among 442 urothelial bladder carcinoma patients. The impact of pathologic stage, margin status, nodal involvement, and extent of node dissection on failure patterns was assessed using competing risk analysis. We calculated the percentage of patients whose sites of LF would have been completely encompassed within various hypothetical clinical target volumes (CTVs) for postoperative radiation. Results: Compared with stage {<=}pT2, stage {>=}pT3 patients had higher 5-year LF rates in virtually all pelvic sites. Among stage {>=}pT3 patients, margin status significantly altered the failure pattern whereas extent of node dissection and nodal positivity did not. In stage {>=}pT3 patients with negative margins, failure occurred predominantly in the iliac/obturator nodes and uncommonly in the cystectomy bed and/or presacral nodes. Of these patients in whom failure subsequently occurred, 76% would have had all LF sites encompassed within CTVs covering only the iliac/obturator nodes. In stage {>=}pT3 with positive margins, cystectomy bed and/or presacral nodal failures increased significantly. Only 57% of such patients had all LF sites within CTVs limited to the iliac/obturator nodes, but including the cystectomy bed and presacral nodes in the CTV when margins were positive increased the percentage of LFs encompassed to 91%. Conclusions: Patterns of failure within the pelvis are summarized to facilitate design of adjuvant RT protocols. These data suggest that RT should target at least the iliac/obturator nodes in stage {>=}pT3 with negative margins; coverage of the presacral nodes and cystectomy bed may be necessary for stage {>=}pT3 with positive margins.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

A prediction model of failure threshold for shear deformation in a Zr-based bulk metallic glass

The failure of bulk metallic glasses (BMGs) during plastic deformation at room temperature is abrupt and instantaneous, while the analysis of precursor information based on avalanche events helps predict catastrophic failure. An acoustic emission (AE) signal can provide accurate precursor information for material failure, due to its sensitive and high fast calculation ability. In the current study, AE monitoring tests are carried out during uniaxial compression tests of BMGs at different strain rates. The AE experimental failure threshold, E max , is proposed on the basis of AE cumulative energy, which reflects the intensity of damage evolution at different loading conditions. Compared with the critical shear band velocity (CSBV) associated with stick-slip dynamics of serrated flow, E max is a more sensitive failure parameter since it is connected with the local microscopic changes that occur during the material response process. Here, the E max is obtained prior to reaching the CSBV since the calculation of these two avalanches analysis focuses on the different stages of shear band growth. In particular, AE events are related to the “dry” friction process in the first stage, however, the CSBV is responsible for the “viscous” glide in the second stage. Therefore, Emax is not affected by the complex interactions between the shear bands during the stick-slip process. The maximum avalanche of serrated flow, S max , is proposed as the experimental failure threshold, which depends on the applied strain rate as S max ~ $\dot{ε}$ –λ . According to the relationship of E max and S max , the theoretical failure threshold, E max , follows a criterion E max = 2545$\dot{ε}$ –λ - 4468, where λ is equivalent to 0.15 for this work. Finally, combining the different calculations and AE measurements, this model gives new insights to predict the deformation failure behavior of Zr-based BMGs.

36 MATERIALS SCIENCE↗

Modeling and Simulating Multiple Failure Masking enabled by Local Recovery for Stencil-based Applications at Extreme Scales

By obtaining multi-process hard failure resilience at the application level is a key challenge that must be overcome before the promise of exascale can be fully realized. Some previous work has shown that online global recovery can dramatically reduce the overhead of failures when compared to the more traditional approach of terminating the job and restarting it from the last stored checkpoint. If online recovery is performed in a local manner further scalability is enabled, not only due to the intrinsic lower costs of recovering locally, but also due to derived effects when using some application types. In this paper we model one such effect, namely multiple failure masking, that manifests when running Stencil parallel computations on an environment when failures are recovered locally. First, the delay propagation shape of one or multiple failures recovered locally is modeled to enable several analyses of the probability of different levels of failure masking under certain Stencil application behaviors. These results indicate that failure masking is an extremely desirable effect at scale which manifestation is more evident and beneficial as the machine size or the failure rate increase.

97 MATHEMATICS AND COMPUTING↗

SU-D-204-07: Retrospective Correlation of Dose Accuracy with Regions of Local Failure for Early Stage Lung Cancer Patients Treated with Stereotactic Body Radiotherapy

Purpose: To correlate dose distributions computed using six algorithms for recurrent early stage non-small cell lung cancer (NSCLC) patients treated with stereotactic body radiotherapy (SBRT), with outcome (local failure). Methods: Of 270 NSCLC patients treated with 12Gyx4, 20 were found to have local recurrence prior to the 2-year time point. These patients were originally planned with 1-D pencil beam (1-D PB) algorithm. 4D imaging was performed to manage tumor motion. Regions of local failures were determined from follow-up PET-CT scans. Follow-up CT images were rigidly fused to the planning CT (pCT), and recurrent tumor volumes (Vrecur) were mapped to the pCT. Dose was recomputed, retrospectively, using five algorithms: 3-D PB, collapsed cone convolution (CCC), anisotropic analytical algorithm (AAA), AcurosXB, and Monte Carlo (MC). Tumor control probability (TCP) was computed using the Marsden model (1,2). Patterns of failure were classified as central, in-field, marginal, and distant for Vrecur ≥95% of prescribed dose, 95–80%, 80–20%, and ≤20%, respectively (3). Results: Average PTV D95 (dose covering 95% of the PTV) for 3-D PB, CCC, AAA, AcurosXB, and MC relative to 1-D PB were 95.3±2.1%, 84.1±7.5%, 84.9±5.7%, 86.3±6.0%, and 85.1±7.0%, respectively. TCP values for 1-D PB, 3-D PB, CCC, AAA, AcurosXB, and MC were 98.5±1.2%, 95.7±3.0, 79.6±16.1%, 79.7±16.5%, 81.1±17.5%, and 78.1±20%, respectively. Patterns of local failures were similar for 1-D and 3D PB plans, which predicted that the majority of failures occur in centraldistal regions, with only ∼15% occurring distantly. However, with convolution/superposition and MC type algorithms, the majority of failures (65%) were predicted to be distant, consistent with the literature. Conclusion: Based on MC and convolution/superposition type algorithms, average PTV D95 and TCP were ∼15% lower than the planned 1-D PB dose calculation. Patterns of failure results suggest that MC and convolution/superposition type algorithms predict different outcomes for patterns of failure relative to PB algorithms. Work supported in part by Varian Medical Systems, Palo Alto, CA.

60 APPLIED LIFE SCIENCES↗

Failure Rate Estimates for Passive Mechanical Components

Fusion design projects that reach the stage of design reviews typically require reliability, availability, maintainability and inspectability (RAMI) assessments. While many kinds of reliability and maintainability data are needed for a wide variety of fusion components to quantify a RAMI assessment, one of the types of components requiring data is the passive mechanical component. Passive mechanical components can be items such as metal plates, rods, shafts, brackets, bolts, and other items. Generally these passive components tend to have low failure rates since they are simple designs, have no or few moving parts, and do not need control signals to perform their design function. However, there are no widely recognized datasets for such components, so analysts must use multiple datasets and judgment for failure rates. This report gives recommended failure rates for various failure modes of these components, and gives suggested failure rate multipliers to apply to these failure rates if they reside in harsh environments. If the component resides far from the tokamak then typical industrial failure rates would suffice. Components near the tokamak find conditions can be demanding; similar to those in a fission reactor containment building. Some component failure rates can be modified if the components reside inside the tokamak.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Failure Analysis for Molten Salt Thermal Energy Storage Tanks for In-Service CSP Plants

Thermal Energy Storage (TES) is a fundamental component in concentrating solar power (CSP) plants to increase the plant's dispatchability, capacity factor, while reducing the levelized cost of electricity. In central receivers CSP plants, nitrate molten salts have been used for several years for operation temperatures of up to 565 degrees C. Despite many efforts to advance nitrate salt to higher operation temperatures (even considering a replacement with molten chloride salts) to achieve higher energy conversion efficiencies, the 565 degrees C temperature is currently considered the state-of-the art. Although molten salt tanks have been broadly deployed in commercial CSP plants worldwide, several failures have been reported in these tanks after a few months or years of operation, causing significant economic loss and mistrust in CSP technologies. Most of these failures are associated with the infancy of the technology and multiple issues related to tank design, fabrication, commissioning, and aggressive operation. A technical standard dedicated to the design and fabrication of molten nitrate TES tanks does not exist today. Current in-service molten salt tanks have been generally designed based on the American Petroleum Institute's (API) 650 and ASME Section II standards. The API 650 code provides guidelines for dimensions and fabrication for oil storage tanks up to 260 degrees C. The ASME standard provides allowable stress values for various materials at a range of temperatures and conditions. Both standards seem to be limited for molten salt TES tanks where high temperatures, thermal cycling, and transient conditions are expected. In 2020, NREL released the Concentrating Solar Power Best Practices Study (NREL/TP-5500-75763) that summarized multiple issues in CSP plants, along with potential alternatives and recommendations to address those issues based on information collected from participants representing about 80% of operating CSP plants in the world. One of the recommendations from this study was the development of accurate and validated models to evaluate the plant's transient operation, capable of capturing the effect of short-term clouds and operator response, while being flexible in being adapted to various spatial and temporal resource data. The "Failure Analysis for Molten Salt Thermal Energy Tanks for In-Service CSP Plants" project was inspired on this recommendation and was focused on (1) the development and validation of a physics-based model for a representative, commercial-scale molten salt tank, (2) performing simulations to evaluate the behavior of the tank as a function of typical plant operation conditions, (3) understanding tank failures mechanisms, (4) determining the residual stress and distortion in the tank floor after welding fabrication and evaluating their impact in the stresses developed in the tank during operation, (5) assessing the impact of key operation parameters on the temperature and stress distribution, (6) conduct a preliminary evaluation of design features to reduce stress and improve tank's reliability, and (7) estimate the tank's service life based on the stress developed under diverse operation scenarios. From the analysis conducted in the project and presented in this report, it was found that maximum stresses surpassing the yield strength point of the stainless steel (SS) 347H are developed on the tank floor near the perimeter. These large stresses are strongly influenced by the initial residual stresses and distortion of the tank floor after welding fabrication. During operation, large stresses are developed in the tank floor at high operation temperatures with large salt inventory levels during transient operation. High stresses are also related to elevated temperature gradients in the tank floor that could be attributed to insufficient mixing within the salt inflow and the salt inventory. Based on the analysis, creep is the predominant failure mechanism. However, the large stress levels could favor the plastic deformation into buckles, and crack formation due to stress relaxation cracking during cycle operation. A lifetime below 3 years was estimated for the typical plant operation conditions and a specific initial residual stress and deformation distribution of the tank floor. The estimated life agrees with the service time to failure reported in several commercial molten salt tanks. Desing and operation guidelines can be extracted from the analysis presented in this report, which could be adopted by tank manufacturers and CSP operators to advance toward an ultimate solution for tank failures by reducing residual and operational stresses to achieve a tank service life of more than 30 years. Addressing failures in molten salt TES tanks is fundamental for the CSP industry's survivability, but it is also important for other industrial and power generation applications using this technology, including nuclear and concentrating solar thermal.

14 SOLAR ENERGY↗

Analyzing Potential Failures and Effects in a Pilot-Scale Biomass Preprocessing Facility for Improved Reliability

This study demonstrates a failure identification methodology applied to a preprocessing facility generating conversion-ready feedstocks from biomass meeting conversion process critical quality attribute (CQA) specifications. Failure Modes and Effects Analysis (FMEA) was used as an industrially relevant risk analysis approach to evaluate a logging residue preprocessing system to prepare feedstock for pyrolysis conversion. Risk evaluations considered both system-level and operation unit-level assessments considering process efficiency, product quality, cost, sustainability, and safety. Key outputs included estimations of semi-quantitative risk scores for each failure, identification of the failure impacts, identification of failure causes associated with material attributes and process parameters, ranking success rates of failure detection methods, and speculation of potential mitigation strategies for decreasing failure risk scores. Results showed that deviations from moisture specifications had cascading consequences for other CQAs along with process safety implications. Failures linked to fixed carbon specifications carried the highest risk scores for product quality and process efficiency impacts. As increased throughput can be inversely related to meeting product quality specifications; achieving throughput and other material-based CQAs simultaneously will likely require system optimization or prioritization based on system economics. Ultimately, this work successfully demonstrates FMEA as a risk analysis approach for other bioenergy process systems.

09 BIOMASS FUELS↗

Regional Nodal Involvement and Patterns of Spread Along In-Transit Pathways in Children With Rhabdomyosarcoma of the Extremity: A Report From the Children's Oncology Group;Rhabdomyosarcoma; Regional failure; In-transit nodes; Radiotherapy; Extremity

Purpose: To evaluate the incidence and prognostic factors for regional failure, with attention to the in-transit pathways of spread, in children with nonmetastatic rhabdomyosarcoma of the extremity. Methods and Materials: The Intergroup rhabdomyosarcoma studies III, IV-Pilot, and IV enrolled 226 children with rhabdomyosarcoma of the extremity. Failure at the in-transit (epitrochlear/brachial and popliteal) and proximal (axillary/infraclavicular and inguinal/femoral) lymph nodes was evaluated. The median follow-up for the surviving patients was 10.4 years. Results: Of the 226 children, 55 (24%) had clinical or pathologic evidence of either in-transit and/or proximal lymph node involvement at diagnosis. The actuarial 5-year risk of regional failure was 12%. The prognostic factors for poor regional control were female gender and lymph node involvement at diagnosis. In the 116 patients with a distal extremity primary tumor, 5% had in-transit lymph node involvement at diagnosis. The estimated 5-year incidences of in-transit and proximal nodal failure was 12% and 8%, respectively. The in-transit failure rate was 0% for patients who underwent radiotherapy and/or underwent lymph node sampling of the in-transit nodal site but was 15% for those who did not (p = .07). However, the 5-year event-free survival rate did not differ between these two groups (64% vs. 55%, respectively, p = .47). Conclusion: The high incidence of regional involvement necessitates aggressive identification and treatment of regional lymph nodes in patients with rhabdomyosarcoma of the extremity. In patients with distal extremity tumors, in-transit failures were as common as failures in more proximal regional sites. Patients who underwent complete lymph node staging with appropriate radiotherapy to the in-transit nodal site, if indicated, were at a slightly lower risk of in-transit failure.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Failure Rates and Patterns of Recurrence in Patients With Resected N1 Non-Small-Cell Lung Cancer

Purpose: To examine the local and distant recurrence rates and patterns of failure in patients undergoing potentially curative resection of N1 non-small-cell lung cancer. Methods and Materials: The study included 60 consecutive unirradiated patients treated from 2000 to 2006. Median follow-up was 30 months. Failure rates were calculated by the Kaplan-Meier method. A univariate Cox proportional hazard model was used to assess factors associated with recurrence. Results: Local and distant failure rates (as the first site of failure) at 2, 3, and 5 years were 33%, 33%, and 46%; and 26%, 26%, and 32%, respectively. The most common site of local failure was in the mediastinum; 12 of 18 local recurrences would have been included within proposed postoperative radiotherapy fields. Patients who received chemotherapy were found to be at increased risk of local failure, whereas those who underwent pneumonectomy or who had more positive nodes had significantly increased risks of distant failure. Conclusions: Patients with resected non-small-cell lung cancer who have N1 disease are at substantial risk of local recurrence as the first site of relapse, which is greater than the risk of distant failure. The role of postoperative radiotherapy in such patients should be revisited in the era of adjuvant chemotherapy.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Void coalescence and ductile failure in IN718 investigated via high-energy synchrotron X-ray tomography and diffraction

Ductile failure through the growth and coalescence of voids is of particular relevance for many engineering materials. Yet, the lack of experimental measurements of the mechanical state of the material at an appropriate length scale has limited further understanding of the regime of ductile failure. In this study, local grain-scale experimental measurements are used to determine their relevance for describing and predicting ductile failure. Additive manufactured materials, due to the presence of inherent porosity, as well as the ability to tailor internal porosity, provides a promising avenue to study ductile failure. Selective laser melting is used to manufacture a specially designed specimen with two large, internal voids, in addition to the natural porosity, which is characteristic to the production process. The initial porosity and its evolution upon tensile loading are characterized via micro tomography. Several locations of void coalescence are captured in the sample, revealing activity of multiple modes of failure. Finite element simulations, with a simplified J2 plasticity model, instantiated with the initial void structure, is deployed, in which geometric localizations of stress concentration corresponded with experimentally observed sites of coalescence but was inadequate in capturing the appropriate failure mechanism. Furthermore, from the experimental results of the far-field high-energy diffraction microscopy, the heterogeneous micromechanical state is identified and tracked around voids due to the local grain interactions. These experiments determine narrow bands of low stress triaxiality, at the onset of failure, which highlight the path of coalescence through intervoid shearing. Diffraction spot spreading analysis aided characterization of intragranular plasticity and strain heterogeneity, which can be coupled with high triaxiality to form the conditions inducive for coalescence through intervoid necking. In this study, the experimentally determined grain-scale description of the micromechanical state provides a physical basis that can accurately capture both the sites and the mechanism of void coalescence at the onset of ductile failure.

36 MATERIALS SCIENCE↗

Emergence of Diverse Failure Patterns in Weathering‐Induced Landslides: Insights From Particle Finite Element Simulations

Weathering is a fundamental driver of landslide evolution over geological timescales. Despite its ubiquity and importance, quantifying how weathering drives the progressive destabilization of rock slopes remains challenging. In this work, we develop a unified computational framework based on the particle finite element method to investigate the evolution of weathering‐induced landslides, from long‐term weathering to short‐term slope failure and runout dynamics. The framework integrates key processes, including weathering front propagation, time‐dependent strength degradation, rupture surface development, and post‐failure runout dynamics. Through numerical simulation experiments, we elucidate how interactions among weathering characteristics (type, intensity, and rate law), bedrock strength, fracture distribution, and slope geometry govern the failure modes and kinematics of weathering‐induced landslides. Simulations show that matrix‐dominated weathering leads to shallow translational failures, whereas fracture‐dominated weathering produces deep‐seated rotational and compound landslides. Pre‐existing fractures and slope morphology also strongly influence the movement of destabilized landmasses, affecting the failure pattern (e.g., kinematic mode and rupture surface geometry) and post‐failure behavior (e.g., runout velocity). We further demonstrate that the failure time and volume of weathered slopes are governed by the competition between gravitational driving forces and cohesive resisting forces during progressive destabilization. These findings provide new insights into the fundamental mechanisms that drive the emergence of diverse failure patterns of weathering‐induced landslides with important implications for landslide hazard assessment.

Wang, Liang [Eidgenoessische Technische Hochschule↗

SU-F-T-246: Evaluation of Healthcare Failure Mode And Effect Analysis For Risk Assessment

Purpose: To evaluate the differences between the Veteran Affairs Healthcare Failure Modes and Effect Analysis (HFMEA) and the AAPM Task Group 100 Failure and Effect Analysis (FMEA) risk assessment techniques in the setting of a stereotactic radiosurgery (SRS) procedure were compared respectively. Understanding the differences in the techniques methodologies and outcomes will provide further insight into the applicability and utility of risk assessments exercises in radiation therapy. Methods: HFMEA risk assessment analysis was performed on a stereotactic radiosurgery procedure. A previous study from our institution completed a FMEA of our SRS procedure and the process map generated from this work was used for the HFMEA. The process of performing the HFMEA scoring was analyzed, and the results from both analyses were compared. Results: The key differences between the two risk assessments are the scoring criteria for failure modes and identifying critical failure modes for potential hazards. The general consensus among the team performing the analyses was that scoring for the HFMEA was simpler and more intuitive then the FMEA. The FMEA identified 25 critical failure modes while the HFMEA identified 39. Seven of the FMEA critical failure modes were not identified by the HFMEA and 21 of the HFMEA critical failure modes were not identified by the FMEA. HFMEA as described by the Veteran Affairs provides guidelines on which failure modes to address first. Conclusion: HFMEA is a more efficient model for identifying gross risks in a process than FMEA. Clinics with minimal staff, time and resources can benefit from this type of risk assessment to eliminate or mitigate high risk hazards with nominal effort. FMEA can provide more in depth details but at the cost of elevated effort.

60 APPLIED LIFE SCIENCES↗

An Experimental Study of Shear-Dominated Failure in the 2013 Sandia Fracture Challenge Specimen

This report presents an experimental study motivated by results obtained during the 2013 Sandia Fracture Challenge. The challenge involved A286 steel, shear-dominated compression specimens whose load-deflection response contained a load maximum fol- lowed by significant displacement under decreasing load, ending with a catastrophic fracture. Blind numerical simulations deviated from the experiments well before the maximum load and did not predict the failure displacement. A series of new tests were conducted on specimens machined from the original A286 steel stock to learn more about the deformation and failure processes in the specimen and potentially improve future numerical simulations. The study consisted of several uniaxial tension tests to explore anisotropy in the material, and a set of new tests on the compression speci- men. In some compression specimen tests, stereo digital image correlation (DIC) was used to measure the surface strain fields local to the region of interest. In others, the compression specimen was loaded to a given displacement prior to failure, unloaded, sectioned, and imaged under the microscope to determine when material damage first appeared and how it spread. The experiments brought the following observations to light. The tensile tests revealed that the plastic response of the material is anisotropic. DIC during the shear- dominated compression tests showed that all three in-plane surface strain components had maxima in the order of 50% at the maximum load. Sectioning of the specimens revealed no signs of material damage at the point where simulations deviated from the experiments. Cracks and other damage did start to form approximately when the max- imum load was reached, and they grew as the load decreased, eventually culminating in catastrophic failure of the specimens. In addition to the steel specimens, a similar study was carried out for aluminum 7075-T651 specimens. These specimens achieved much lower loads and displacements, and failure occurred very close to the maximum in the load-deflection response. No material damage was observed in these specimens, even when failure was imminent. In the future, we plan to use these experimental results to improve numerical simu- lations of the A286 steel experiments, and to improve plasticity and failure models for the Al 7075 stock. The ultimate goal of our efforts is to increase our confidence in the results of numerical simulations of elastic-plastic structural behavior and failure.

36 MATERIALS SCIENCE↗

Holistic Measurement Driven Resilience: Combining Operational Fault and Failure Measurements and Fault Injection for Quantifying Fault Detection, Propagation and Impact. Final report

For HPC systems to date, application resilience to faults and failures has been accomplished by the brute- force method of checkpoint/restart, which allows an application to make forward progress in the face of system and application faults, errors, and failures independent of root cause or end result. It has remained the primary resilience mechanism because we lack a way to identify faults and anticipate consequences early enough to take meaningful mitigating action. However, checkpoint/restart implementations put a tremendous burden on system resources and on the applications themselves and is becoming less feasible at scale. Because we have not yet operated at scales at which checkpoint/restart fails to provide forward progress, despite increasing costs, vendors have had little motivation to provide the instrumentation necessary for early identification of faults and failures. However, as we move from petascale to exascale, component mean time to failure (MTTF) will render the existing techniques ineffectual and/or too expensive. Furthermore, fault recovery mechanisms such as failover and/or error correction introduce performance inconsistency. Instrumentation allowing early indication of problems and tools to enable use of such information by systems, operating systems, and applications offer an alternative, more scalable and less costly solution. In the HMDR project, we built on our experience and expertise developed and accumulated over years of research on design, monitoring, measurement, and assessment of resilient computing systems. Analysis of field data on the current and past generations of extreme-scale systems revealed several challenges that, if not addressed in increasingly larger and more complex systems, may hinder the effectiveness of future exascale computing systems. Specifically, i) file systems and interconnects in current-generation large-scale systems already operate at the margins of resiliency, including consistent performance, and may not scale to larger deployments; ii) automated, software-based failover mechanisms are frequently inadequate and can introduce wider failures, such that failures during recovery may lead to system/application failures, including system-wide outages; and iii) silent data corruption represents a critical fault mode and will require efficient detection mechanisms if next-generation applications are to take full advantage of exascale hardware. To address the above challenges, we assembled a team of world-renowned experts in resilient extreme- scale computing from the University of Illinois (Electrical and Computer Engineering, Computer Science, and NCSA), SNL, LANL, NERSC, and Cray. Our team includes representatives from centers that house many of the largest HPC resources in the world, both today and over the coming years. The team has a unique track record of research in i) system and application failure characterization based on the analysis of field data, ii) data-driven design of fault/error detection mechanisms, and iii) experimental characterization of system/application resiliency. The team includes system owners/operators who provide continuous data collection and access and ensure installation of appropriate analysis tools.

97 MATHEMATICS AND COMPUTING↗

Quantifying effectiveness of failure prediction and response in HPC systems : methodology and example.

Effective failure prediction and mitigation strategies in high-performance computing systems could provide huge gains in resilience of tightly coupled large-scale scientific codes. These gains would come from prediction-directed process migration and resource servicing, intelligent resource allocation, and checkpointing driven by failure predictors rather than at regular intervals based on nominal mean time to failure. Given probabilistic associations of outlier behavior in hardware-related metrics with eventual failure in hardware, system software, and/or applications, this paper explores approaches for quantifying the effects of prediction and mitigation strategies and demonstrates these using actual production system data. We describe context-relevant methodologies for determining the accuracy and cost-benefit of predictors. While many research studies have quantified the expected impact of growing system size, and the associated shortened mean time to failure (MTTF), on application performance in large-scale high-performance computing (HPC) platforms, there has been little if any work to quantify the possible gains from predicting system resource failures with significant but imperfect accuracy. This possibly stems from HPC system complexity and the fact that, to date, no one has established any good predictors of failure in these systems. Our work in the OVIS project aims to discover these predictors via a variety of data collection techniques and statistical analysis methods that yield probabilistic predictions. The question then is, 'How good or useful are these predictions?' We investigate methods for answering this question in a general setting, and illustrate them using a specific failure predictor discovered on a production system at Sandia.

99 GENERAL AND MISCELLANEOUS↗

Failure Mode Classification for Life Prediction Modeling of Solid-State Lighting

Since the passing of the Energy Independence and Security Act of 2007, the U.S. government has mandated greater energy independence which has acted as a catalyst for accelerating and facilitating research efforts toward the development and deployment of market-driven solutions for energy-saving homes, buildings and manufacturing, as well as sustainable transportation and renewable electricity generation. As part of this effort, an emphasis toward advancing solid-state lighting technology through research, development, demonstration, and commercial applications is assisting in the phase out of the common incandescent light bulb, as well as developing a more economical lighting source that is less toxic than compact fluorescent lighting. This has led lighting manufacturers to pursue SSL technologies for a wide range of consumer lighting applications. An SSL luminaire’s lifetime can be characterized in terms of lumen maintenance life. Lumen maintenance or lumen depreciation is the percentage decrease in the relative luminous flux from that of the original, pristine luminous flux value. Lumen maintenance life is the estimated operating time, in hours, when the desired failure threshold is projected to be reached at normal operating conditions. One accepted failure threshold of SSL luminaires is lumen maintenance of 70% -- a 30% reduction in the light output of the luminaire. Currently, the only approved lighting standard that puts forth a recommendation for long-term luminous flux maintenance projections towards a specified failure threshold of an SSL luminaire is the IES TM-28-14 (TM28) standard. iii TM28 was derived as a means to compare luminaires that have been tested at different facilities, research labs or companies. TM28 recommends the use of the Arrhenius equation to determine SSL device specific reaction rates from thermally driven failure mechanisms used to characterize a single failure mode – the relative change in the luminous flux output or “light power” of the SSL luminaire. The use of the Arrhenius equation necessitates two different temperature conditions, 25°C and 45°C are suggested by TM28, to determine the SSL lamp specific activation energy. One principal issue with TM28 is the lack of additional stresses or parameters needed to characterize non-temperature dependent failure mechanisms. Another principal issue with TM28 is the assumption that lumen maintenance or lumen depreciation gives an adequate comparison between SSL luminaires. Additionally, TM28 has no process for the determination of acceleration factors or lifetime estimations. Currently, a literature gap exists for established accelerated test methods for SSL devices to assess quality, reliability and durability before being introduced into the marketplace. Furthermore, there is a need for Physics-of-Failure based approaches to understand the processes and mechanisms that induce failure for the assessment of SSL reliability in order to develop generalized acceleration factors that better represent SSL product lifetime. This and the deficiencies in TM28 validate the need behind the development of acceleration techniques to quantify SSL reliability under a variety of environmental conditions. The ability to assess damage accrual and investigate reliability of SSL components and systems is essential to understanding the life time of the SSL device itself. The methodologies developed in this work increases the understanding of SSL devices iv through the investigation of component and device reliability under a variety of accelerated test conditions. The approaches for suitable lifetime predictions through the development of novel generalized acceleration factors, as well as a prognostics and health management framework, will greatly reduce the time and effort needed to produce SSL acceleration factors for the development of lifetime predictions.

Sakalaukus, Peter Joseph↗

Life Prediction and Classification of Failure Modes in Solid State Luminaires Using Bayesian Probabilistic Models

A new method has been developed for assessment of the onset of degradation in solid state luminaires to classify failure mechanisms by using metrics beyond lumen degradation that are currently used for identification of failure. Luminous Flux output, Correlated Color Temperature Data on Philips LED Lamps has been gathered under 85°C/85%RH till lamp failure. The acquired data has been used in conjunction with Bayesian Probabilistic Models to identify luminaires with onset of degradation much prior to failure through identification of decision boundaries between lamps with accrued damage and lamps beyond the failure threshold in the feature space. In addition luminaires with different failure modes have been classified separately from healthy pristine luminaires. It is expected that, the new test technique will allow the development of failure distributions without testing till L70 life for the manifestation of failure.

Lall, Pradeep↗

Mini-Ckpts: Surviving OS Failures in Persistent Memory

Concern is growing in the high-performance computing (HPC) community on the reliability of future extreme-scale systems. Current efforts have focused on application fault-tolerance rather than the operating system (OS), despite the fact that recent studies have suggested that failures in OS memory are more likely. The OS is critical to a system's correct and efficient operation of the node and processes it governs -- and in HPC also for any other nodes a parallelized application runs on and communicates with: Any single node failure generally forces all processes of this application to terminate due to tight communication in HPC. Therefore, the OS itself must be capable of tolerating failures. In this work, we introduce mini-ckpts, a framework which enables application survival despite the occurrence of a fatal OS failure or crash. Mini-ckpts achieves this tolerance by ensuring that the critical data describing a process is preserved in persistent memory prior to the failure. Following the failure, the OS is rejuvenated via a warm reboot and the application continues execution effectively making the failure and restart transparent. The mini-ckpts rejuvenation and recovery process is measured to take between three to six seconds and has a failure-free overhead of between 3-5% for a number of key HPC workloads. In contrast to current fault-tolerance methods, this work ensures that the operating and runtime system can continue in the presence of faults. This is a much finer-grained and dynamic method of fault-tolerance than the current, coarse-grained, application-centric methods. Handling faults at this level has the potential to greatly reduce overheads and enables mitigation of additional fault scenarios.

Fiala, David↗