SEARCH · Engineering Papers
Results for “data analytic”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Sensor Data Analytics and Data Quality Assessment Software
The proposed framework derives a set of quality metrics to provide critical insights into and tracking of grid operations, sensor performance, sensor longevity, and event statistics. Power grid engineers can utilize this information to identify problems with existing sensor locations and problematic power grid assets including generators, transmission lines, load centers, and substations. This information can also be used to identify unexpected/abnormal behavior of power grid components, improve power grid observability, and operational monitoring, and thus enhance real-time decision-making support system. Power grid planners can utilize this information to augment existing sensing architecture with new sensors and improve the observability of the network.
Autonomy Loops for Monitoring, Operational Data Analytics, Feedback, and Response in HPC Operations
Many High Performance Computing (HPC) facilities have developed and deployed frameworks in support of continuous monitoring and operational data analytics (MODA) to help improve efficiency and throughput. Because of the complexity and scale of systems and workflows and the need for low-latency response to address dynamic circumstances, automated feedback and response have the potential to be more effective than current human-in-the-loop approaches which are laborious and error prone. Progress has been limited, however, by factors such as the lack of infrastructure and feedback hooks, and successful deployment is often site- and case-specific. In this position paper we report on the outcomes and plans from a recent Dagstuhl Seminar, seeking to carve a path for community progress in the development of autonomous feedback loops for MODA, based on the established formalism of similar (MAPE-K) loops in autonomous computing and self-adaptive systems. By defining and developing such loops for significant cases experienced across HPC sites, we seek to extract commonalities and develop conventions that will facilitate interoperability and interchangeability with system hardware, software, and applications across different sites, and will motivate vendors and others to provide telemetry interfaces and feedback hooks to enable community development and pervasive deployment of MODA autonomy loops.
Challenges for monitoring and data analytics in a leadership public data repository
The availability and disposition of data has assumed increasing importance in large-scale computational science. Data repositories are evolving to meet new classes of requirements: compliance with government access guidelines, support for reproducibility of experimental results, and long-term availability of data products. The Constellation public data repository at the Oak Ridge Leadership Computing Facility faces these issues while being situated in one of the most productive data centers in the world. While monitoring and operational data analysis are ingrained in the operation of the OLCF’s large-scale high performance computing platforms, data repositories do not have this history of support. Problems faced by Constellation range from data size (over 7 petabytes in current holdings) to analytic complexity (detailed curation is both absolutely necessary for many data sets and absolutely impossible for humans to accomplish in any practical manner) to deployment environment (OLCF storage resources are oriented toward the needs of the compute platforms). In this paper we describe some of the challenges for collecting monitoring and analytic data from a leadership public data repository. We also discuss various strategies we are pursuing in order to address these challenges, from manual data collection to plans for introducing machine learning-based curatorial techniques.
An Integrated Platform for Collaborative Data Analytics
While collaboration among data scientists is a key to organizational productivity, data analysts face significant barriers to achieving this end, including data sharing, accessing and configuring the required computational environment, and a unified method of sharing knowledge. Each of these barriers to collaboration is related to the fundamental question of knowledge management “how can organizations use knowledge more effectively?”. In this paper, we consider the problem of knowledge management in collaborative data analytics and present ShareAL, an integrated knowledge management platform, as a solution to that problem. The ShareAL platform consists of three core components: a full stack web application, a dashboard for analyzing streaming data and a High Performance Computing (HPC) cluster for performing real time analysis. Prior research has not applied knowledge management to collaborative analytics or developed a platform with the same capabilities as ShareAL. ShareAL overcomes the barriers data scientists face to collaboration by providing intuitive sharing of data and analytics via the web application, a shared computing environment via the HPC cluster and knowledge sharing and collaboration via a real time messaging application.
Metall: A persistent memory allocator for data-centric analytics
Data analytics applications transform raw input data into analytics-specific data structures before performing analytics. Unfortunately, such data ingestion steps are often more expensive than analytics. In addition, various types of NVRAM devices are already used in many HPC systems today. Such devices will be useful for storing and reusing data structures beyond a single process life cycle. We developed Metall, a persistent memory allocator built on top of the memory-mapped file mechanism. Metall enables applications to transparently allocate custom C++ data structures into various types of persistent memories. Metall incorporates a concise and high-performance memory management algorithm inspired by Supermalloc and the rich C++ interface developed by Boost.Interprocess library. On a dynamic graph construction workload, Metall achieved up to 11.7x and 48.3x performance improvements over Boost.Interprocess and memkind (PMEM kind), respectively. We also demonstrate Metall’s high adaptability by integrating Metall into a graph processing framework, GraphBLAS Template Library. Here this study’s outcomes indicate that Metall will be a strong tool for accelerating future large-scale data analytics by allowing applications to leverage persistent memory efficiently.
DAAP (Data Analytics Application Profiling)
The Data Analytics Application Profiling (DAAP) utility is a software program comprised of a library and scripts to provide basic instrumenting of parallel scientific software applications while they are running on an HPC cluster, and transmit this data over a secure connection as individual records to aggregators. When used in conjunction with additional open source software components (Telegraf, FluentD, and RabbitMQ), the data collected by DAAP can be transmitted off-cluster to an analytics system of the user’s choice for further analysis.
High-dimensional data analytics in civil engineering: A review on matrix and tensor decomposition
Recent developments in sensing and monitoring techniques have led to the generation of high-dimensional data in the field of civil engineering. High-dimensional data analytics methods have thus been developed to interpret such complex data. Among the different high-dimensional data analytics techniques, matrix and tensor decomposition methods have acquired a notable interest in the civil engineering community over the past decade. Due to their unique ability to deal with highly redundant and correlated data, these methods are establishing themselves as promising and efficient tools to analyze high-dimensional data in the civil engineering arena. In this paper, high-dimensional data is referred to as a data set in which the number of features is comparable or larger than the number of observations. This review paper aims to summarize the applications of matrix and tensor decomposition methods in civil engineering over the last decade. The survey begins with a general overview of matrix and tensor decomposition followed by highlighting their significance in the field. Afterward, various applications of these high-dimensional data analytics methods in civil engineering are presented, while the advantages offered by these methods are discussed. Lastly, challenges and potential research avenues for employing matrix and tensor decomposition and future emerging trends for their novel use are highlighted.
COTS Data Analytics Software User Manual: Version 1.0
Large volumes of data are being collected by Sandia National Laboratories as part of an active commercial-off-the-shelf (COTS) part testing and surveillance program. This user manual documents Python-based COTS Data Analytics software that has been developed for standardizing, displaying, visualizing, and analyzing the resulting COTS part testing and surveillance data. It is the objective of these software tools to streamline the analysis of COTS testing and surveillance data and improve the efficiency with which test engineers and data analytics experts can pinpoint possible performance and reliability problems in COTS parts.
Application of Systems Engineering Principles and Techniques in Biological Big Data Analytics: A Review
In the past few decades, we have witnessed tremendous advancements in biology, life sciences and healthcare. These advancements are due in no small part to the big data made available by various high-throughput technologies, the ever-advancing computing power, and the algorithmic advancements in machine learning. Specifically, big data analytics such as statistical and machine learning has become an essential tool in these rapidly developing fields. As a result, the subject has drawn increased attention and many review papers have been published in just the past few years on the subject. Different from all existing reviews, this work focuses on the application of systems, engineering principles and techniques in addressing some of the common challenges in big data analytics for biological, biomedical and healthcare applications. Specifically, this review focuses on the following three key areas in biological big data analytics where systems engineering principles and techniques have been playing important roles: the principle of parsimony in addressing overfitting, the dynamic analysis of biological data, and the role of domain knowledge in biological data analytics.
Materials Characterization, Prediction and Control Project: Summary Report on Data Analytics Framework
This report summarizes the activities performed under the data analytics Vertex in the Materials Characterization, Prediction and Control Project funded under laboratory directed research and development at Pacific Northwest National Laboratory. The data analytics Vertex developed models for associating global or local process parameters, microstructural features, and performance properties of friction-stir-processed 316L stainless steel plates. Statistical, machine learning, and deep learning models, as well as generative artificial intelligence approaches, were used to develop the associations between the process-structure-property data streams. These associations formed the basis for predicting global properties of parts manufactured under different process envelopes, providing a basis for predicting performance using data driven as well as physics-informed and physics-constrained approaches. Additionally, the associations were used to predict local process parameters and microstructural features of the product, predictive relationships that have the potential to form the basis of a control framework that could eventually modulate a friction-stir process to maintain product quality.
Analysis of Air-Purifying Respirator (APR) and Powered Air-Purifying Respirator (PAPR) Cartridge Performance Testing on a Hanford AP Tank Farm Exhauster Slipstream: Volume 2 Raw Analytical Data
As the Tank Operations Contractor for U.S. Department of Energy operations at the Hanford site in Washington State, Washington River Protection Solutions (WRPS) is responsible for managing highly radioactive wastes stored in tanks at Hanford. WRPS tests air-purifying respirator (APR) and powered air-purifying respirator (PAPR) chemical cartridges commonly used at Hanford Tank Farms to determine the period of time that the cartridges would provide adequate performance for APRs and PAPRs used to protect workers when exposed to a mixture of Chemicals of Potential Concern (COPC) from any vapors exiting headspaces in the storage tanks. The Occupational Safety and Health Administration (OSHA) Standard promulgated in Title 29 of the Code of the Federal Regulations (CFR) 1910.134(d)(3)(iii)(b)(2) specifies that for protection against gases and vapors, employers shall implement a schedule for cartridges to ensure that change-outs occur before the end of service life. The change schedule can be based on objective information or data that ensures cartridge change-outs occur before the end of their service life.[2-5] The primary function of the WRPS Cartridge Test Program is to obtain objective data to determine service lives for the APR and PAPR cartridges used at Hanford Tank Farms. WRPS contracted with Pacific Northwest National Laboratory to analyze the test data and offer an independent analysis and any recommendations. Volume 1 of this report summarizes data analysis of APR and PAPR cartridge testing on vapors from the AP tank farm exhauster. Previous testing of APR cartridges was conducted on the AP exhauster in June 2016. However, an AP exhauster upgrade was completed in September 2016. Two different APR cartridges from SCOTT Safety (Monroe, North Carolina) were assessed for the new AP exhauster source, along with two different PAPR cartridges—one from MSA Safety Inc. (Pittsburgh, Pennsylvania) and another from 3M (Maplewood, Minnesota). Volume 2 provides an introduction to the raw data, including analytical laboratory analysis results that supported the analysis and conclusions documented in Volume 1.
Analysis of Air-Purifying Respirator (APR) Cartridge Performance Testing on Hanford Tanks SX-101 and SX-104: Volume 2 - Raw Analytical Data
As the Tank Operations Contractor for U.S. Department of Energy operations at the Hanford site in Washington State, Washington River Protection Solutions (WRPS) is responsible for managing highly radioactive wastes stored in tanks at Hanford. To protect workers at Hanford Tank Farms, WRPS tests air-purifying respirator (APR) and powered air-purifying respirator (PAPR) chemical cartridges commonly used at the tank farms. The tests were conducted to determine the period of time the cartridges would provide adequate performance for APRs and PAPRs when workers are exposed to a mixture of Chemicals of Potential Concern (COPC) from any vapors exiting headspaces in the storage tanks. Occupational Safety and Health Administration (OSHA) Standard 29 Code of the Federal Regulations (CFR) 1910.134(d)(3)(iii)(b)(2) specifies that for protection against gases and vapors, employers shall implement a schedule for cartridges to ensure that change-outs occur before the end of service life. The change schedule can be based on objective information or data that ensures cartridge change-outs occur before the end of their service life. The primary function of the WRPS Cartridge Test Program is to obtain objective data to determine service lives for the APR and PAPR cartridges used at Hanford Tank Farms. WRPS contracted with Pacific Northwest National Laboratory to analyze the test data and offer an independent analysis and any recommendations. This report summarizes data analysis of APR cartridge testing on headspace vapors from Hanford SX-101 and SX-104 single-shell tanks. Volume 1 of this report documents the testing, data analysis, results, conclusions, and recommendations resulting from the APR testing on SX-101 and SX-104 headspace vapors. Volume 2 provides an introduction to the raw data, including analytical laboratory analysis results that supported the analysis and conclusions documented in Volume 1.
Analysis of Air-Purifying Respirator (APR) and Powered Air-Purifying Respirator (PAPR) Cartridge Performance Testing on Hanford Tanks BY-108 and BY-110 Volume 2: Raw Analytical Data
As the Tank Operations Contractor for U.S. Department of Energy operations at the Hanford site in Washington State, Washington River Protection Solutions (WRPS) is responsible for managing highly radioactive wastes stored in tanks at Hanford. WRPS tests air-purifying respirator (APR) and powered air-purifying respirator (PAPR) chemical cartridges commonly used by workers at Hanford Tank Farms to determine the period of time that the cartridges would provide adequate performance for APRs and PAPRs used to protect workers when exposed to a mixture of Chemicals of Potential Concern (COPC) from any vapors exiting headspaces in the storage tanks. The Occupational Safety and Health Administration (OSHA) Standard promulgated in Title 29 of the Code of the Federal Regulations (CFR) 1910.134(d)(3)(iii)(b)(2) specifies that for protection against gases and vapors, employers shall implement a schedule for cartridges to ensure that change-outs occur before the end of service life. The change schedule can be based on objective information or data that ensures cartridge change-outs occur before the end of their service life. The primary function of the WRPS Cartridge Test Program is to obtain objective data to determine service lives for the APR and PAPR cartridges used at Hanford Tank Farms. WRPS contracted with Pacific Northwest National Laboratory to analyze the test data and offer an independent analysis and any recommendations. This report summarizes data analysis of APR and PAPR cartridge testing on BY-110 and BY-108 headspace vapors, respectively. Two different APR cartridges from SCOTT Safety (Monroe, North Carolina) were assessed for the BY-110 headspace vapors, and two different PAPR cartridges—one from MSA Safety Inc. (Pittsburgh, Pennsylvania) and another from 3M (Maplewood, Minnesota)—were assessed for the BY-108 headspace vapors. Volume 1 of this report documents the testing, data analysis, results, conclusions, and recommendations resulting from the cartridge testing of vapors from the BY-110 and BY-108 headspaces. Volume 2 provides an introduction to the raw data, including analytical laboratory analysis results that supported the analysis and conclusions documented in Volume 1.
Analysis of Air-Purifying Respirator (APR) and Powered Air-Purifying Respirator (PAPR) Cartridge Performance Testing on a Hanford AX Tank Farm Exhauster Slipstream Volume 2: Raw Analytical Data
As the Tank Operations Contractor for U.S. Department of Energy operations at the Hanford site in Washington State, Washington River Protection Solutions (WRPS) is responsible for managing highly radioactive wastes stored in tanks at Hanford. To protect workers at Hanford Tank Farms, WRPS tests air-purifying respirator (APR) and powered air-purifying respirator (PAPR) chemical cartridges commonly used at the tank farms. The tests were conducted to determine the period of time the cartridges would provide adequate performance for APRs and PAPRs when workers are exposed to a mixture of Chemicals of Potential Concern (COPC) from any vapors exiting headspaces in the storage tanks. Occupational Safety and Health Administration (OSHA) Standard 29 Code of the Federal Regulations (CFR) 1910.134(d)(3)(iii)(b)(2) specifies that for protection against gases and vapors, employers shall implement a schedule for cartridges to ensure that change-outs occur before the end of service life. The change schedule can be based on objective information or data that ensures cartridge change-outs occur before the end of their service life. The primary function of the WRPS Cartridge Test Program is to obtain objective data to determine service lives for the APR and PAPR cartridges used at Hanford Tank Farms. WRPS contracted with Pacific Northwest National Laboratory to analyze the test data and offer an independent analysis and any recommendations. This report summarizes data analysis of APR and PAPR cartridge testing on a vapor slipstream from the Hanford AX tank farm exhauster. Volume 1 of this report documents the testing, data analysis, results, conclusions, and recommendations resulting from the cartridge testing on AX exhauster slipstream vapors. Volume 2 provides an introduction to the raw data, including analytical laboratory analysis results that supported the analysis and conclusions documented in Volume 1.
Analysis of Powered Air-Purifying Respirator (PAPR) Cartridge Performance Testing on Hanford Tanks SX-101 and SX-104: Volume 2 Raw Analytical Data
As the Tank Operations Contractor for U.S. Department of Energy operations at the Hanford site in Washington State, Washington River Protection Solutions (WRPS) is responsible for managing highly radioactive wastes stored in tanks at Hanford. WRPS tests of cartridges used in air-purifying respirators (APR) and powered air-purifying respirators (PAPR) commonly used at Hanford Tank Farms. The tests are conducted to determine the period of time the cartridges would provide adequate performance for APRs and PAPRs used to protect workers when exposed to a mixture of Chemicals of Potential Concern (COPC) from any vapors exiting headspaces in the storage tanks. Occupational Safety and Health Administration (OSHA) Standard 29 Code of the Federal Regulations (CFR) 1910.134(d)(3)(iii)(b)(2) specifies that for protection against gases and vapors, employers shall implement a schedule for cartridges to ensure that change-outs occur before the end of service life. The change schedule can be based on objective information or data that ensures cartridge change-outs occur before the end of their service life. The primary function of the WRPS Cartridge Test Program is to obtain objective data to determine service lives for the APR and PAPR cartridges used at Hanford Tank Farms. WRPS contracted with Pacific Northwest National Laboratory to analyze the test data and offer an independent analysis and any recommendations. This report summarizes data analysis of PAPR cartridge testing on headspace vapors from Hanford SX-101 and SX-104 single-shell tanks. Two different PAPR cartridges—one from MSA Safety Inc. (Pittsburgh, Pennsylvania) and another from 3M (Maplewood, Minnesota)—were assessed on each tank headspace source on separate days. These data represent the first PAPR cartridge testing under the recent WRPS program, as testing to date had been focused on APR cartridges. Volume 1 of this report documents the testing, data analysis, results, conclusions, and recommendations resulting from the PAPR testing on SX-101 and SX-104 headspace vapors. Volume 2 provides an introduction to the raw data, including analytical laboratory analysis results that supported the analysis and conclusions documented in Volume 1.
Open-Source Data for MAC-POSTS: Mobility Data Analytics Center - Prediction, Optimization, and Simulation Toolkit for Transportation Systems
MAC-POSTS (Mobility Data Analytics Center - Prediction, Optimization, and Simulation toolkit for Transportation Systems) is a toolkit for dynamic transportation network modeling. Developed by the Mobility Data Analytics Center (MAC) at Carnegie Mellon University, this package implements many classic dynamic transportation network models, as well as new models proposed by MAC members. It has served as one building block for many other models and research projects. As such, this package used to be treated as an internal research project of the MAC lab, and admittedly, the code base is messy, and the interface is hard to use. However, we are working hard to make it a generally usable and useful toolkit for dynamic transportation network modeling. We would really appreciate any feedback, comments, suggestions, or criticisms.