Health Assessment and Performance Monitoring of Large Machine Diagnostics
Explore the source record for details and available documents.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Explore the source record for details and available documents.
Explore the source record for details and available documents.
Explore the source record for details and available documents.
Using DTW for HPC / LDMS data clustering
Performance evaluation and analysis of large scale computing applications is essential for optimal use of resources. As detector simulation is one of the most compute intensive tasks and Geant4 is the simulation toolkit most widely used in contemporary high energy physics (HEP) experiments, it is important to monitor Geant4 through its development cycle for changes in computing performance and to identify problems and opportunities for code improvements. All Geant4 development and public releases are being profiled with a set of applications that utilize different input event samples, physics parameters, and detector configurations. Results from multiple benchmarking runs are compared to previous public and development reference releases to monitor CPU and memory usage. Observed changes are evaluated and correlated with code modifications. Besides the full summary of call stack and memory footprint, a detailed call graph analysis is available to Geant4 developers for further analysis. The set of software tools used in the performance evaluation procedure, both in sequential and multi-threaded modes, include FAST, IgProf and Open|Speedshop. In conclusion, the scalability of the CPU time and memory performance in multi-threaded application is evaluated by measuring event throughput and memory gain as a function of the number of threads for selected event samples.
Washington River Protections Solutions, LLC (WRPS) is the prime contractor for the Department of Energy (DOE) Tank Farm Operations at the Hanford Site, and is responsible for the management and retrieval of millions of gallons of radioactive and chemical wastes, stored in 177 underground tanks; a legacy of decades of production activities at the Site. The potential for workforce exposure to the chemical vapors from the tanks has long been a concern for the workforce and for DOE and its tank contractors; concerns about tank vapors is an issue that pre-dates WRPS's incumbency as the Tank Operations Contractor. In response to an increase in the number of tank vapor events reported by workers at the Hanford Site in 2014, WRPS commissioned the Savannah River National Laboratory with establishing and overseeing a panel of external, independent experts (Tank Vapor Assessment Team (TVAT)) to examine chemical vapor management and related worker-protection measures at the Hanford Tank Farms. The TVAT released the Hanford Tank Vapor Assessment Report (SRNL-RP-2014-00791) in late 2014, which included several recommendations to reduce the potential for chemical vapor exposures. From 2015- 2018, governmental and external groups conducted additional assessments to review the progress made in implementing the TVAT recommendations, and the overall Industrial Hygiene (IH) program. All of these assessments specifically recommended WRPS take action to improve communications with the workforce around the subject of tank vapors. Some further recommended assessing the effectiveness of vapors-related communications. In fiscal year 2016, WRPS and DOE Office of River Protection (DOE-ORP) formed an Integrated Project Team (IPT) to: (1) develop the strategy for addressing chemical vapors and (2) define the future chemical vapor protection actions at the Tank Farms. The actions are designed to address the recommendations from the external assessments as part of the overall effort to mitigate vapors-related exposure incidences, and to ensure workers are safe and feel safe while working in and around the Hanford Tank Farms. A comprehensive vapor management communication plan, engagement processes, and effectiveness measurements were established as essential components of the IPT effort. The Chemical Protection Program Office (CPPO) was formed, in part, to deliver on these IPT initiatives, monitor performance, and drive improvements to WRPS's vapors-related communications and engagement efforts at the Hanford Tank Farms. The CPPO was created as a temporary program office tailored to address these unique challenges by augmenting traditional company performance monitoring and communications (i.e. all employee e-mails, posters, and formal briefings). Modeled in broad concept after the successful program office WRPS created to support the American Recovery and Reinvestment Act, CPPO has also proven successful in its mission. Because concerns about chemical vapors in the workplace were expressed by the workforce, they are a key stakeholder. As such, the CPPO has concentrated its information sharing efforts toward the WRPS workforce. With a focus on empathetic engagement, continuous improvement, and transparent information sharing, the CPPO has developed and implemented a strategy for improving worker trust in vapors-related communications and increasing the workforce knowledge of responses taken to address chemical vapors. Effective implementation of this strategy is demonstrated by the results of workforce surveys, which indicate measured and important improvement in the effectiveness of vapors-communications and the knowledge level workers report about vapor issues compared to the previous year. This paper discusses the utilization of a temporary program office and the unique approach that the CPPO implemented to achieve success, including: strategic application of the technical and communications staff skill mix of the organization (including talent development opportunities such as stretch assignments for WRPS's Strategic Talent Development program members and support from WRPS college interns), teaming with other WRPS' organizations such as IH, Engineering and Communications and Public Relations (C and PR), the development of new types of information sharing to support the effort, and the methods by which the effectiveness of the communications efforts are measured to drive continuous improvement. (authors)
Pacific Northwest National Laboratory is conducting a treatability test designed to demonstrate that in situ biostimulation can be applied to help meet cleanup goals in the Hanford Site 100-D Area. The in situ biostimulation technology is intended to provide supplemental treatment upgradient of the In Situ Redox Manipulation (ISRM) barrier by reducing the concentration of the primary oxidizing species in groundwater (i.e., nitrate and dissolved oxygen) and chromate, and thereby increasing the longevity of the ISRM barrier. This report summarizes the initial results from field testing of an in situ biological treatment zone implemented through injection of a soluble substrate. The field test is divided into operational phases that include substrate injection, process monitoring, and performance monitoring. The results summarized herein are for the substrate injection and process monitoring phase encompassing the first approximately three months of field testing. Performance monitoring is ongoing at the time this report was prepared and is planned to extend over approximately 18 months. As such, this report is an interim data summary report for the field test. The treatability testing has multiple objectives focused on evaluating the performance of biostimulation as a reducing barrier for nitrate, oxygen, and chromate. The following conclusions related to these objectives are supported by the data provided in this report. Substrate was successfully distributed to a radius of about 15 m (50 ft) from the injection well. Monitoring data indicate that microbial growth initiated rapidly, and this rapid growth would limit the ability to inject substrate to significantly larger zones from a single injection well. As would be expected, the uniformity of substrate distribution was impacted by subsurface heterogeneity. However, subsequent microbial activity and ability to reduce the targeted species was observed throughout the monitored zone during the process monitoring period, and low nitrate and oxygen concentrations were maintained. Chromate concentrations in the treatment zone began to increase about two months after substrate injection, up to about 30 percent of the background concentration upgradient of the test site. The performance monitoring phase will provide additional data to interpret the performance of the biostimulation process and information for scale-up as a reducing barrier.
This report details the progress made on the ASCR funded project Performance Health Monitoring for Large Scale Systems. A large-scale application may not achieve its full performance potential due to degraded performance of even a single subsystem. Detecting performance faults, isolating them, and taking remedial action is critical for the scale of systems on the horizon. PHM aims to develop techniques and tools that can be used to identify and mitigate such performance problems. We accomplish this through two main aspects. The PHM framework encompasses diagnostics, system monitoring, fault isolation, and performance evaluation capabilities that indicates when a performance fault has been detected, either due to an anomaly present in the system itself or due to contention for shared resources between concurrently executing jobs. Software components called the PHM Control system then build upon the capabilities provided by the PHM framework to mitigate degradation caused by performance problems.
Part of the promise of exascale computing and the next generation of scientific simulation codes is the ability to bring together time and spatial scales that have traditionally been treated separately. This enables creating complex coupled simulations and in situ analysis pipelines, encompassing such things as "whole device" fusion models or the simulation of cities from sewers to rooftops. Unfortunately, the HPC analysis tools that have been built up over the preceding decades are ill suited to the debugging and performance analysis of such computational ensembles. In this paper, we present a new vision for performance measurement and understanding of HPC codes, MonitoringAnalytics (MONA). MONA is designed to be a flexible, high performance monitoring infrastructure that can perform monitoring analysis in place or in transit by embedding analytics and characterization directly into the data stream, without relying upon delivering all monitoring information to a central database for post-processing. It addresses the trade-offs between the prohibitively expensive capture of all performance characteristics and not capturing enough to detect the features of interest. We demonstrate several uses of MONA; capturing and indexing multi-executable performance profiles to enable later processing, extraction of performance primitives to enable the generation of customizable benchmarks and performance skeletons, and extracting communication and application behaviors to enable better control and placement for the current and future runs of the science ensemble. Relevant performance information based on a system for MONA built from ADIOS and SOSflow technologies is provided for DOE science applications and leadership machines.
BGQNCL is a library to monitor and record network performance counters on the 5D torus interconnection network of IBM's Blue Gene/Q platform.
This report updates a previous report covering the performance of a permeable reactive barrier installed at 100N. In this report we re-evaluate the results after having an additional year of performance monitoring data to incorporate.
FLOPS (FLoating-point Operations Per Second) is a commonly used performance metric for scientific programs that rely heavily on floating-point (FP) calculations. The metric is based on the number of FP operations rather than instructions, thereby facilitating a fair comparison between different machines. A well-known use of this metric is the LINPACK benchmark that is used to generate the Top500 list. It measures how fast a computer solves a dense N by N system of linear equations Ax=b, which requires a known number of FP operations, and reports the result in millions of FP operations per second (MFLOPS). While running a benchmark with known FP workloads can provide insightful information about the efficiency of a machine's FP pipelines in relation to other machines, measuring FLOPS of an arbitrary scientific application in a platform-independent manner is nontrivial. The goal of this paper is twofold. First, we explore the FP microarchitectures of key processors that are underpinning the LC machines. Second, we present the hardware performance monitoring counter-based measurement techniques that a user can use to get the native FLOPS of his or her program, which are practical solutions readily available on LC platforms. By nature, however, these native FLOPS metrics are not directly comparable across different machines mainly because FP operations are not consistent across microarchitectures. Thus, the first goal of this paper represents the base reference by which a user can interpret the measured FLOPS more judiciously.
This document presents a conceptual design for a contingency interim measure (IM) for treatment of the public water supply system at Barnes, Kansas, should this become necessary. The aquifer that serves the public water supply system at Barnes has been affected by trace to low concentrations of carbon tetrachloride and its degradation product, chloroform. Investigations conducted on behalf of the CCC/USDA by Argonne National Laboratory (Argonne 2008a) have demonstrated that groundwater at the Barnes site is contaminated with carbon tetrachloride at concentrations exceeding the Kansas Tier 2 risk-based screening level (RBSL) and the EPA maximum contaminant level (MCL) of 5.0 {micro}g/L for this compound. The Commodity Credit Corporation of the U.S. Department of Agriculture (CCC/USDA) formerly operated a grain storage facility in Barnes, approximately 800 ft east-southeast of the public water supply wells. Carbon tetrachloride was used in the treatment of grain. Another potential source identified in an investigation conducted for the KDHE (PRC 1996) is the site of a former agriculture building owned by the local school district (USD 223). This building is located immediately east of well PWS3. The potential contingency IM options evaluated in this report include the treatment of groundwater at the public water supply wellheads and the provision of an alternate water supply via Washington County Rural Water District No.2 (RWD 2). This document was developed in accordance with KDHE Bureau of Environmental Remediation (BER) Policy No.BER-RS-029 (Revised) (KDHE 2006a), supplemented by guidance from the KDHE project manager. Upon the approval of this contingency IM conceptual design by the KDHE, the CCC/USDA will prepare a treatment system design document that will contain the following elements: (1) Description of the approved contingency IM treatment method; (2) Drawings and/or schematics provided by the contractor and/or manufacturer of the approved technology; (3) A detailed work schedule with a generic timetable; (4) A detailed operations and maintenance plan; (5) A proposed performance monitoring plan; and (6) A fact sheet associated with a community relations plan.
This chapter provides background information on why sustaining operations of combined cooling, heating and power systems is important, provides the algorithms for CHP system performance monitoring and commissioning verification, and concludes with a discussion on how these algorithms can be deployed.
BGQNCL is a library to monitor and record network performance counters on the 5D torus interconnection network of IBM's Blue Gene/Q platform.
Strategic Environmental Assessment (SEA) requires monitoring in order to identify unforeseen adverse effects and to enable appropriate remedial action to be taken. Guidelines on how to monitor significant environmental impacts have been developed but experience from practice is limited. This paper presents a study of environmental monitoring in Swedish regional transport infrastructure planning. The result shows that essentially no environmental monitoring is currently performed. Monitoring of the plans merely involves checking the implementation of projects and performing an economic account. At present, a new planning period has commenced for the regional transport infrastructure plans. To obtain an iterative SEA process for the new plan with integrated SEA monitoring, the following means are suggested: reinforcement of practitioners' incentives to plan and perform monitoring; integration of monitoring in the SEA process; pre-determined impact thresholds that prompt remedial action; and more efficient use of monitoring results.
Network Monitoring requires a substantial use of data and error analysis to overcome issues with clusters. Zenoss and Splunk help to monitor system log messages that are reporting issues about the clusters to monitoring services. Infiniband infrastructure on a number of clusters upgraded to ibmon2. ibmon2 requires different filters to report errors to system administrators. Focus for this summer is to: (1) Implement ibmon2 filters on monitoring boxes to report system errors to system administrators using Zenoss and Splunk; (2) Modify and improve scripts for monitoring and administrative usage; (3) Learn more about networks including services and maintenance for high performance computing systems; and (4) Gain a life experience working with professionals under real world situations. Filters were created to account for clusters running ibmon2 v1.0.0-1 10 Filters currently implemented for ibmon2 using Python. Filters look for threshold of port counters. Over certain counts, filters report errors to on-call system administrators and modifies grid to show local host with issue.