Incremental update techniques for online analysis of streaming tensor data
Explore the source record for details and available documents.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explore the source record for details and available documents.
Explore the source record for details and available documents.
Critical infrastructure systems such as the electric grid are increasingly cyber-physical; yet, despite the cyber-physical characteristics of critical infrastructure systems, the physical process system and communication/control network system are traditionally analyzed in siloes. As these systems become more cyber-physical, it is crucial that models and methods are available to assess the cyber physical system (CPS) interdependencies, characteristics, and event propagation for improved planning, operation, and response. Thus, we proposed an integrated structural and temporal CPS interdependency analysis framework, InterGraph-CPS, that provides insight into the CPS function during normal operation as well as disturbances. This integrated structural and temporal interdependency framework is uniquely designed for assessing CPSs by account for the challenges of analyzing cyber and physical data streams together due to data availability, data type, and time scale differences. By leveraging both structural (e.g., graph analysis) and temporal (e.g., data analytics) techniques, different CPS behaviors and configurations can be accounted for.
Adaptive signal processing algorithms that rely upon representations of signal and noise subspaces often require updates to those representations when new data become available. Subspace representations frequently are estimated from available data with singular value (SVD) decompositions. Subspace updates require modifications to these decompositions. Updates can be performed inexpensively provided they are low-rank. A substantial literature on SVD updates exists, frequently focusing on rank-1 updates (see e.g. [Karasalo, 1986; Comon and Golub, 1990, Badeau, 2004]). In these methods, data matrices are modified by addition or deletion of a row or column, or data covariance matrices are modified by addition of the outer product of a new vector. A recent paper by Brand [2006] provides a general and efficient method for arbitrary rank updates to an SVD. The purpose of this note is to describe a closely-related method for applications where right singular vectors are not required. This note also describes the SVD updates to a particular scenario of interest in seismic array signal processing. The particular application involve updating the wideband subspace representation used in seismic subspace detectors [Harris, 2006]. These subspace detectors generalize waveform correlation algorithms to detect signals that lie in a subspace of waveforms of dimension d {ge} 1. They potentially are of interest because they extend the range of waveform variation over which these sensitive detectors apply. Subspace detectors operate by projecting waveform data from a detection window into a subspace specified by a collection of orthonormal waveform basis vectors (referred to as the template). Subspace templates are constructed from a suite of normalized, aligned master event waveforms that may be acquired by a single sensor, a three-component sensor, an array of such sensors or a sensor network. The template design process entails constructing a data matrix whose columns contain the master event waveform data, then performing a singular value decomposition on the data matrix to extract an orthonormal basis for the waveform suite. The template typically is comprised of a subset of the left singular vectors corresponding to the larger singular values. The application involves updating a subspace template when new data become available, i.e. when new defining events are detected for a particular source. It often is the case that the waveforms corresponding to a particular source drift over time [Harris, 2001]. The Green's functions describing propagation can be altered because of changes in the source region. For example, if the source is a mine, signals from explosions may change gradually as a pit is extended (the source moves) or the scattering topography is altered by excavation. This motivates a tracking adjustment to a subspace representation. This note also comments on SVD updates for a related problem. In realistic pipeline operations it often is the case that data from one or more channels of an array are unusable (dead channels, channels with prolific dropouts, etc.). In such cases it is desirable to modify an array subspace template to operate on data only from the remaining usable channels. Furthermore, it is desirable to modify the templates directly without recourse to the original data matrix. Usually the template design process is separate from the application of the template in a detector to a continuous data stream. Consequently, the original data matrix may not be available for template modification at detector run time.
Explore the source record for details and available documents.
The invention combines radiation detection instrumentation, data processing algorithms, and visualization software to enable, for the first time ever, simultaneous, real-time imaging of both gamma ray and neutron sources (fast and slow). Radiation detectors sensitive to gamma rays and neutrons are integrated with readout electronics that allow discriminated particles to be incorporated into real-time 3D volumetric reconstructions as two separate data streams. Using the Scene Data Fusion (SDF) conceptual framework and contextual sensors from the Localization and Mapping Platform (LAMP) developed at LBNL, gamma-ray and neutron data are fused onto contextual information from the environment around the detection system. The invention is compatible with radiation detection media capable of detecting and discriminating between gamma rays and neutrons or the pairing of multiple detectors separately capable of detecting gamma rays and/or neutrons. It can be integrated into a compact, lightweight imaging platform for operation as a hand-carry device, on unmanned aerial systems, and with unmanned vehicles. The ability to measure neutron enables the detection and 3D localization of shielded nuclear materials such as Pu-239 which is not possible with means of gamma-ray detection and spectroscopy overcoming of major limitation in the detection of such materials to-date.
Abstract not provided.
Nowadays, data generated by large-scale scientific experiments are on the scale of petabytes per month. These data are transferred through dedicated high-bandwidth networks across distributed collaboration sites for processing, storage, and analysis. Like general purpose networks, research networks experience attacks. However, monitoring traffic in such high-bandwidth network environments is challenging with conventional deep packet inspection (DPI)- based network intrusion detection systems (NIDS). Recently, machine learning (ML) algorithms have shown potential in detecting anomalous patterns with better generalizability and faster speed. Compared to the general purpose networks, traffic movement in research network is less volatile and more likely to benefit from the ML-based anomaly analysis. We present ThunderSecure, an efficient AI-powered system for detecting anomalies in 100G research networks. ThunderSecure implements a high-throughput packet processing infrastructure using multi-cores and GPUs. It takes time-series statistics from a variety of network measurements and uses 2D Convolutional Neural Network (CNN) to explore both spatial and temporal dependencies in network data stream. Patterns of science data traffic are learned with a one-class neural network, which blends an Adversarial Autoencoder (AAE) with Gaussian mixture density estimation. Testing traffic flows exhibiting significant deviations from the learned baseline of normality are marked as anomalies. We trained ThunderSecure on hundreds of billions of science data packets recorded from a 100G research network at Fermi National Accelerator Laboratory. The detection performance was evaluated on traffic captured from the same research network days and weeks after the training with different types of attack flows injected. To the best of our knowledge, this is the first ML work in the area of network anomaly detection that has been validated on such extreme scale datasets. Results show that ThunderSecure can recognize science data traffic that are captured long after the training and make nearly certain detection on those with anomalous flows injected, even in the case when the anomaly-to-normal mixing ratio is 0.1%.
Explore the source record for details and available documents.
Efficiently processing time series data streams in real-time on resource-constrained devices offers significant advantages in terms of enhanced computational energy efficiency and reduced time-related risks. We introduce an innovative streaming time series classification network that utilizes attentive power iteration, enabling real-time processing on resource-constrained devices. Our model continuously updates a compact representation of the entire time series, enhancing classification accuracy while conserving energy and processing time. Notably, it excels in streaming scenarios without requiring complete time series access, enabling swift decisions. Experimental results show that our approach excels in classification accuracy and energy efficiency, with over 70% less consumption and threefold faster task completion than benchmarks. This work advances real-time responsiveness, energy conservation, and operational effectiveness for constrained devices, contributing to optimizing various applications.
In this paper, we develop a method which we call OnlineGCP for computing the Generalized Canonical Polyadic (GCP) tensor decomposition of streaming data. GCP differs from traditional canonical polyadic (CP) tensor decompositions as it allows for arbitrary objective functions which the CP model attempts to minimize. This approach can provide better fits and more interpretable models when the observed tensor data is strongly non-Gaussian. In the streaming case, tensor data is gradually observed over time and the algorithm must incrementally update a GCP factorization with limited access to prior data. In this work, we extend the GCP formalism to the streaming context by deriving a GCP optimization problem to be solved as new tensor data is observed, formulate a tunable history term to balance reconstruction of recently observed data with data observed in the past, develop a scalable solution strategy based on segregated solves using stochastic gradient descent methods, describe a software implementation that provides performance and portability to contemporary CPU and GPU architectures and integrates with Matlab for enhanced usability, and demonstrate the utility and performance of the approach and software on several synthetic and real tensor data sets.
A high frame rate detector system is described that enables fast real-time data analysis of scanning diffraction experiments in scanning transmission electron microscopy (STEM). This is an end-to-end development that encompasses the data producing detector, data transportation, and real-time processing of data. The detector will consist of a central pixel sensor that is surrounded by annular silicon diodes. Both components of the detector system will synchronously capture data at almost 100 kHz frame rate, which produces an approximately 400 Gb/s data stream. Low-level preprocessing will be implemented in firmware before the data is streamed from the National Center for Electron Microscopy (NCEM) to the National Energy Research Scientific Computing Center (NERSC). Live data processing, before it lands on disk, will happen on the Cori supercomputer and aims to present scientists with prompt experimental feedback. This online analysis will provide rough information of the sample that can be utilized for sample alignment, sample monitoring and verification that the experiment is set up correctly. Only a compressed version of the relevant data is then selected for more in-depth processing.
Nowadays, data generated by large-scale scientific experiments are on the scale of petabytes per month. These data are transferred through dedicated high-bandwidth networks (40/100G) across distributed sites for processing, storage, and analysis. Like general purpose networks, research networks experience intrusions. However, monitoring anomalies in such high-speed network traffics is challenging given current cyber-infrastructure. Moreover, traditional network intrusion detection systems (NIDS) are signature based. However, anomaly patterns are difficult to define and that rulesets are often not updated frequently enough to reflect the changes of attack behaviors. We present ThunderSecure, a high-throughput, unsupervised learning-based intrusions detection system for 100G research networks. ThunderSecure implements an efficient packet processing and detection pipeline using multi-cores and GPUs. It extracts statistical and temporal features from real-time network data streams and feeds them to a one-class anomaly detection network. A baseline of normal distribution will be created based on the training observation. Testing traffic deviated from the learned profile will be marked as anomalies. We trained ThunderSecure on hundreds of billions of science data packets mirrored from two 100G network connections at Fermi National Accelerator Laboratory. The detection performance was evaluated on traffic captured from the same research network days and weeks after the training with different types of attack flows injected. Results show that ThunderSecure can recognize science data traffic captured long after the training and made nearly certain detection on the segment of the streams where anomalous flows were injected.
The growing demand for real-time analysis, experimental steering, and decision-making in scientific workflows has created a need for tightly coupled integrations between experimental facilities and high-performance computing (HPC) systems. The Department of Energy’s Integrated Research Infrastructure (IRI) initiative highlights data streaming as a key capability for enabling memory-to-memory data transfers, bypassing the limitations of traditional store-and-forward models. SciStream is a toolkit developed by researchers at Argonne National Laboratory (ANL) to support such streaming by addressing cross-domain security, delegated authentication, and application transparency. We deployed and evaluated SciStream on the Oak Ridge Leadership Computing Facility’s (OLCF) Advanced Computing Ecosystem (ACE) infrastructure, leveraging the Olivine OpenShift cluster and its high-bandwidth Data Streaming Nodes (DSNs) as gateway nodes. Our evaluation included synthetic streaming workloads derived from IRI science workflows, a streaming simulator, and integration with RabbitMQ to handle low-level messaging. This report documents the deployment process, performance evaluation, and challenges encountered, along with opportunities for future improvements.
The nuclear industry is experiencing a steady increase in maintenance costs even though plants are maintained under high levels of safety, capability and reliability. Nuclear power plants are expected to run every unit at maximum capacity at all times, efficiently utilizing assets with minimal downtime. Surveillance and maintenance of nuclear decommissioning infrastructure provides lot challenges with respect to maintenance or decommissioning of the buildings. There is a need for a framework to analyze the huge amount of data generated by the sensors on the nuclear reactor components as well as structures, to monitor the conditions of these building over a period of time. Emerging technologies such as big data analytics have become a requirement in the nuclear industry to improve structural health monitoring and diagnostics. FIU will make use of existing mature technologies in the areas of imaging, robotics, big data, and machine learning/deep learning to assess the structural integrity of aging facilities at DOE sites. As these facilities await decommissioning, there is a need to understand the structural health of these structures. Many of these facilities were built over 50 years ago and in some cases these facilities have gone beyond the operational life expectancy. In other cases, the facilities have been placed in a state of 'cold and dark' and they are sitting unused, awaiting decommissioning. Finally, some older facilities are one-of-a-kind operational/production facilities supporting critical DOE missions and cannot be shut down. In any of these scenarios, the structural integrity of these facilities may be compromised, so it is imperative that adequate inspections and data collection/analysis be performed on a continuous and ongoing basis. The primary goals of the research include collecting various formats of data such as structured, unstructured and streaming data from the various sensors deployed in buildings, collect videos/pictures from various imaging sources, ingest them using a Hadoop distributed file system and process the data using Spark to perform batch processing and real time analytics. Research and development on various machine learning/deep learning algorithms will also be performed to analyze the heterogeneous data collected from nuclear decommission infrastructure. FIU will design the big data framework to ingest, store and process huge amounts of heterogeneous data collected from many sources and optimize the algorithms to provide insights into the data and predict anomalies observed when compared against baseline conditions. Various modules of this framework will include heterogeneous data sources, message broker, Hadoop distributed file system, Spark for stream and batch processing, machine learning/ deep learning, Cassandra - persistent data store and visualization. (authors)
We report the rate of data generated by cutting-edge experimental science facilities and large-scale simulations enabled by current high-performance computing (HPC) systems has continued to grow at a far greater pace than the development of the network and storage capabilities on which these systems rely. To cope with this challenge, scientist are moving toward the creation of autonomous experiments and HPC simulations using machine learning. However, efficiently moving, storing, and processing large amounts of data away from the point of origin presents an incredible challenge. In-memory computing, in situ analysis, data staging, and data streaming are recognized viable alternatives to traditional file-based methods for transferring data between coupled workflows. However, the performance trade-offs and limitations for these methods are not fully understood when used in HPC applications. This article presents a comprehensive performance assessment of the current solutions for data staging when applied to applications that are not necessary I/O intensive which makes them not ideal candidates for these methods. Our study is based on experiments running at scale on Oak Ridge National Laboratory's Summit supercomputer using applications and simulations that cover typical computational motifs and patterns. We investigated the usability and cost/benefit trade-offs of staging algorithms for HPC applications under different scenarios and highlight opportunities for optimizing the dataflow between coupled simulation workflows.
Abstract Metal-based additive manufacturing requires active monitoring solutions for assessing part quality. Multiple sensors and data streams, however, generate large heterogeneous data sets that are impractical for manual assessment and characterization. In this work, an automated pipeline is developed that enables feature extraction from high-speed camera video and multi-modal data analysis. The framework removes the need for manual assessment through the utilization of deep learning techniques and training models in a weakly supervised paradigm. We demonstrate this pipeline’s capability over 700,000 high-speed camera frames. The pipeline successfully extracts melt pool and spatter geometries and links them to corresponding pyrometry, radiography, and processparameter information. 715 individual prints are examined to reveal melt pool areas that exceeds 0.07 mm 2 and pyrometry signal over a threshold (375 pyrometry units) were more likely to have defects. These automated processes enable massive throughput of characterization techniques.
This proof-of-concept research was performed to explore the feasibility of using real-world braking data from commercial motor vehicles to make a diagnosis of brake condition similar to that of the performance-based brake tester (PBBT). This was done by determining the relationship between pressure and brake force (P-BF), compensating for the gross vehicle weight (GVW). The nature of this P-BF relationship (e.g., low braking force for a given brake application pressure) may indicate brake system problems. In order to determine the relationship between brake force and brake application pressure, a few key parameters of duty cycle information were collected. Because braking events are often brief, spanning only a few seconds, a sample rate of 10 Hz was needed. The algorithm under development required brake application pressure and speed (from which deceleration was calculated). Accurate weight estimation was also needed to properly derive the braking force from the deceleration. In order to ensure that braking force was the predominant factor in deceleration for the segments of data used in analysis, the data was screened for grade as well. Also, the analysis needed to be based on pressures above the crack pressure. The crack pressure is the pressure below which the individual brakes are not applied due the nature of the mechanical system. This value, which may vary somewhat from one wheel end to another, is approximately 10 psi. Therefore, only pressures 15 psi and above were used in the analysis. The Department of Energy s Medium Truck Duty Cycle research has indicated that under the real-world circumstances of the test vehicle brake pressures of up to approximately 30 psi can be expected. Several different types of data were collected during the testing task of this project. Constant-pressure stopping tests were conducted at several combinations of brake application pressure (15, 20, 25, and 30 psi), load conditions (moderately and fully laden), and speeds (20 and 30 mph). Data was collected at 10 Hz. Standard and stepped-pressure performance-based brake tests with brake pressure transducers were performed for each loading condition. The stepped-pressure test included the constant-pressure intervals of brake application at 15, 20, 25, and 30 psi. The PBBT data files included 10 Hz streaming data collected during the testing of each axle. Two weeks of real-world duty cycle (driving and braking) data was also collected at 10 Hz. Initial analysis of the data revealed that the data collected in the field (i.e., day-to-day operations) provided the same information as that obtained from the controlled tests. Analysis of the data collected revealed a strong linear relationship between brake application pressure and deceleration for given GVWs. As anticipated, initial speed was not found to be a significant factor in the deceleration-pressure relationship, unlike GVW. The positive results obtained from this proof of concept test point to the need for further research to expand this concept. A second phase should include testing over a wider range of speeds and include medium brake application pressures in addition to the low pressures tested in this research. Testing on multiple vehicles would also be of value. This future phase should involve testing to determine how degradation of braking performance affects the pressure-deceleration relationship.