Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 415 records · Page 23

Deep learning at the edge enables real-time streaming ptychographic imaging

Abstract Coherent imaging techniques provide an unparalleled multi-scale view of materials across scientific and technological fields, from structural materials to quantum devices, from integrated circuits to biological cells. Driven by the construction of brighter sources and high-rate detectors, coherent imaging methods like ptychography are poised to revolutionize nanoscale materials characterization. However, these advancements are accompanied by significant increase in data and compute needs, which precludes real-time imaging, feedback and decision-making capabilities with conventional approaches. Here, we demonstrate a workflow that leverages artificial intelligence at the edge and high-performance computing to enable real-time inversion on X-ray ptychography data streamed directly from a detector at up to 2 kHz. The proposed AI-enabled workflow eliminates the oversampling constraints, allowing low-dose imaging using orders of magnitude less data than required by traditional methods.

36 MATERIALS SCIENCE↗

CLARA: A Contemporary Approach to Physics Data Processing

In traditional physics data processing (PDP) systems, data location is static and is accessed by analysis applications. In comparison, CLARA (CLAS12 Reconstruction and Analysis framework) is an environment where data processing algorithms filter continuously flowing data. In CLARA's domain of loosely coupled services, data is not stored, but rather flows from one service to another, mutating constantly along the way. Agents, performing event processing, can then subscribe to particular data/events at any stage of the data transformation, and make intricate decisions (e.g. particle ID) by correlating events from multiple, parallel data streams and/or services. This paper presents a PDP application development framework based on service oriented and event driven architectures. This system allows users to design (Java, C++, and Python languages are supported) and deploy data processing services, as well as dynamically compose PDP applications using available services. The PDP service bus provides a layer on top of a distributed pub-sub middleware implementation, which allows complex service composition and integration without writing code. Examples of service creation and deployment, along with the CLAS12 track reconstruction application design will be presented.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Biologically inspired approaches for biosurveillance anomaly detection and data fusion

This study developed and tested biologically inspired computational methods to detect anomalous signals in data streams that could indicate a pending outbreak or bio-weapon attack. Current large-scale biosurveillance systems are plagued by two principal deficiencies: (1) timely detection of disease-indicating signals in noisy data and (2) anomaly detection across multiple channels. Anomaly detectors and data fusion components modeled after human immune system processes were tested against a variety of natural and synthetic surveillance datasets. A pilot scale immune-system-based biosurveillance system performed at least as well as traditional statistical anomaly detection data fusion approaches. Machine learning approaches leveraging Deep Learning recurrent neural networks were developed and applied to challenging unstructured and multimodal health surveillance data. Within the limits imposed of data availability, both immune systems and deep learning methods were found to improve anomaly detection and data fusion performance for particularly challenging data subsets.

59 BASIC BIOLOGICAL SCIENCES↗

Data Analytics for Electrical Distribution Systems with Micro PMUs (GMLC 1.4.9 Technical Report)

This report describes how electrical utilities with residential, industrial, commercial and government customers make decisions using Data Analytics (DA) applications and discusses how OpenFMB was leveraged to share that data among disparate types of devices for DA. OpenFMB is a data standard developed by utilities for distributed communication and control of power system assets. Here we explored how OpenFMB can be used for sharing analytics and what those analytics could be for a set of representative grid devices. A review of DA applications performed by electrical utilities and customers is presented. An overview of how the testbed we used was built and configured is summarized. Several experiments which were carried out to explore the capabilities of OpenFMB alongside other data sharing mechanisms for gathering sensor measurements and sharing analytics is also discussed. And finally a comparison of OpenFMB and LBNL’s SPARK framework is summarized. Today’s electrical utilities focus on monitoring the real-time customer consumptions, offering new real-time tariff models based on customer usage habits (rewards and punishments), providing energy peak during peak load times (plan energy generation and distribution), improving the load management for transformers and power lines, and controlling the energy and grid resources efficiently. On the other hand, the DA applications for residential customers are based on modifying the customer electricity usage habit to save money, making accurate and timely decisions for electrical billing. However, the DA applications for industrial, commercial and government customers focus on creating efficient programs to improve the environmental condition (reducing CO₂ emission), developing better load forecasting for industrial and/or commercial activities. In this project, a testbed was built to explore how the OpenFMB framework can be used to distribute data among various types of representative grid edge devices. Micro PMUs (Phasor Measurement Unit) were used to collect and transmit data to an OpenPDC server alongside relays with a framework configured with a system using OpenFMB to collect and transmit data. The testbed was composed of a physical asset based on a 60 Hz AC radial power system constructed using SI-GRID components. The power system consisted of a 3-phase 14 volts line-to-ground (24 volts line-to-line) power source, three 3-phase line emulators, a programmable resistive load bank, two smart relays, two micro PMUs, and a (SCADA) and Phasor Data Concentrator (PDC) computer. This testbed configuration allowed us to perform breaker operation tests collecting data from the micro PMUs and from the relays’ sensors. During the micro PMU breaker operation tests we collected the measured and estimated fundamental voltages, currents, frequency as well as the total true, reactive and apparent power. The displacement power factor from two micro PMUs located in parallel power lines was also used. The software steps for the implementation of the Synchrophasor system are explained in detail in this report. The setting (1-2-3) and data collection (4-5) phases were performed with a free download open source software. In the setting phase, the micro PMUs were set (1), the data stream for each micro PMU was validated, and the connection and configuration files were created (2). The PDC computer was set using the connection and configuration files of the micro PMUs (3). In the data collection phase, the binary data from the micro PMUs was collected (4), and the binary data files of the micro PMUs was converted to analyze the test results (5). OpenFMB was configured to use protocol translators to read Modbus and transform the reading into the OpenFMB data model to be read for DA applications.Our experiments showed that by leveraging the OpenFMB data model and the micro PMUs together we could obtain better situational awareness of network faults, which in some cases could lead to failures in the system and would be less perceivable otherwise. For example, in one experiment the fundamental RMS current angle, and the fundamental displacement power factor (estimated by the ratio between the measured real and apparent total power) had a disturbance at the micro PMU connected to the beginning of the parallel power line because the breaker was opened without the operators or SCADA system’s knowledge, and no current was available. Based on the breaker state scenarios and micro PMU locations, the DA applications using the current angle and displacement power factor from the micro PMUs should be studied in detail to avoid a non-desired measurement that could affect the decision from the DA. Also, the sample rates for PMU, Relay and micro PMU devices were compared. The DA application improved the control and operation between the electrical utilities and their customers (residential, industrial, transportation, commercial and government). The higher the sample rate, the better the quality of the measurement for DA estimations.

42 ENGINEERING↗

A digitizer based compact digital spectrometer for ion beam analysis using field programmable gate arrays and various energy algorithms

We report on the implementation of a compact multi-detector fully digital spectrometer and data acquisition system at a nuclear microprobe for ion beam analysis and imaging. The spectrometer design allows for system scalability with no restriction on the number of detectors. It consists of four-channel high-speed digitizer modules for detector signal acquisition and one low-speed digital-to-analog converter (DAC) module with two DAC channels and additional general purpose inputs/outputs to control ion beam scanning and data acquisition. Each digitizer module of the spectrometer provides its own Field Programmable Gate Array (FPGA) as digital signal processing unit to analyze detector signals as well as to synchronize the ion beam position in hard real-time. With the customized FPGA designs for all modules, all calculation intensive tasks are executed inside the modules, which reduces significantly the data stream to and CPU load on the control computer. To achieve an optimal energy resolution for all detector/preamplifier pulse shape characteristics, a user-definable infinite impulse response filter with high throughput for energy determination was implemented. The new spectrometer has an online data analysis feature, a compact size, and is able to process any type of detector signals such as particle induced x-ray emission, Rutherford backscattering spectrometry, or scanning transmission ion microscopy.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Development of a Real-Time, High-Speed Distribution Level Data Acquisition System

With the development of smart grids and the deployment of their enabling technologies, improved data acquisition will be needed at the distribution level to understand the full impact of these changes. With this in mind, NREL has developed a high-speed measurement and data collection network targeted specifically at the distribution level. This network is based around adaptable, rugged measurement devices designed for deployment at a variety of low and medium voltage locations below the sub-station. Each of these devices is capable of real-time data transmission via an Internet connection. Additionally, several analysis and visualization applications have been developed around the incoming data streams.

14 SOLAR ENERGY↗

Performance analysis of data intensive cloud systems based on data management and replication: a survey

As we delve deeper into the ‘Digital Age’, we witness an explosive growth in the volume, velocity, and variety of the data available on the Internet. For example, in 2012 about 2.5 quintillion bytes of data was created on a daily basis that originated from myriad of sources and applications including mobiledevices, sensors, individual archives, social networks, Internet of Things, enterprises, cameras, software logs, etc. Such ‘Data Explosions’ has led to one of the most challenging research issues of the current Information and Communication Technology era: how to optimally manage (e.g., store, replicated, filter, and the like) such large amount of data and identify new ways to analyze large amounts of data for unlocking information. It is clear that such large data streams cannot be managed by setting up on-premises enterprise database systems as it leads to a large up-front cost in buying and administering the hardware and software systems. Therefore, next generation data management systems must be deployed on cloud. The cloud computing paradigm provides scalable and elastic resources, such as data and services accessible over the Internet Every Cloud Service Provider must assure that data is efficiently processed and distributed in a way that does not compromise end-users’ Quality of Service (QoS) in terms of data availability, data search delay, data analysis delay, and the like. In the aforementioned perspective, data replication is used in the cloud for improving the performance (e.g., read and write delay) of applications that access data. Through replication a data intensive application or system can achieve high availability, better fault tolerance, and data recovery. In this paper, we survey data management and replication approaches (from 2007 to 2011) that are developed by both industrial and research communities. The focus of the survey is to discuss and characterize the existing approaches of data replication and management that tackle the resource usage and QoS provisioning with different levels of efficiencies. Moreover, the breakdown of both influential expressions (data replication and management) to provide different QoS attributes is deliberated. Furthermore, the performance advantages and disadvantages of data replication and management approaches in the cloud computing environments are analyzed. Open issues and future challenges related to data consistency, scalability, load balancing, processing and placement are also reported.

Malik, Saif Ur Rehman↗

HERMES

HERMES - High-speed Event Retrieval and Management for Enhanced Spectral imaging code. This code is meant to unpack and process neutron imaging data from the TPX3Cam made by Amsterdam Scientific Instruments. The TPX3Cam utilizes the Timepix3 chip in a single photon counting mode of image acquisition. The photon counting data streaming off the TPX3Cam needs to be process and analyzed in order to create final images. Therefore we are developing both python and cpp codes to allow for users of the TPX3Cam to efficiently analyze data and create images.

Long, Alexander↗

Data Reduction for Science: Brochure from the Advanced Scientific Computing Research Workshop

Data reduction for science holds promise for addressing the challenges of moving, storing, and processing massive data sets produced by the scientific community. Pursuing the PRDs outlined here will enable advances in data streaming, fast feedback and/or autonomous control of experiments, and faster time to scientific insight. These advances will result in a significant improvement in the ability to transport, store, process and interpret experimental, observational, and computational data.

97 MATHEMATICS AND COMPUTING↗

Status of the data acquisition, trigger, and slow control systems of the Mu2e experiment at Fermilab

The Mu2e experiment at the Fermilab will search for a coherent neutrinoless conversion of a muon into an electron in the field of an aluminum nucleus with a sensitivity improvement by a factor of 10,000 over existing limits. In this work, the Mu2e Trigger and Data Acquisition System (TDAQ) uses otsdaq framework as the online Data Acquisition System (DAQ) solution. Developed at Fermilab, otsdaq integrates several framework components — an artdaq-based DAQ, an art-based event processing, and an EPICS-based detector control system (DCS), and provides a uniform multi-user interface to its components through a web browser. Data streams from the Mu2e tracker and calorimeter are handled by the artdaq-based DAQ and processed by a one-level software trigger implemented within the art framework. Events accepted by the trigger have their data combined, post-trigger, with the separately read out data from the Mu2e Cosmic Ray Veto system. Foundation of the Mu2e DCS, EPICS – an Experimental Physics and Industrial Control System – is an open-source platform for monitoring, controlling, alarming, and archiving. A prototype of the TDAQ and the DCS systems has been built and tested over the last three years at Fermilab’s Feynman Computing Center, and now the production system installation is underway. This work presents their status and focus on the installation plans and procedures for racks, workstations, network switches, gateway computers, DAQ hardware, slow controls implementation, and testing.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Unsupervised anomaly clustering via offset alignment in multivariate grid sensing data

Modern industries increasingly rely on multi-sensor technologies to acquire complex, high-dimensional data streams, enabling advanced monitoring and control systems. One critical application is online anomaly detection in electrical smart grids, where multivariate and multimodal sensing technologies play a vital role. However, detecting anomalies in such time-series data is challenging due to their inherent temporal dependencies and stochastic behavior. Traditional approaches based on supervised and semi-supervised learning methods depend on labeled datasets, which are often unavailable in real-world scenarios. While unsupervised methods have emerged as promising alternatives, these methods are highly susceptible to noise and outliers commonly present in sensing applications. Furthermore, deep learning-based anomaly detection methods, despite their performance, are often criticized for their black-box nature, limiting their applicability in safety-critical and online environments where interpretability and explainability are paramount. In this work, we propose an unsupervised anomaly clustering method leveraging a cyclic alignment-based offset detection algorithm for multivariate time-series signals. The proposed method is applied to multivariate data collected from vibrational, voltage, and magnetic field sensors deployed in a local grid substation. Our results demonstrate the robustness of the algorithm in accurately clustering various anomalies/events across different sensing modalities. Additionally, we compare the effectiveness of the proposed approach against a simple pattern-based anomaly detection method, which performs well for univariate data but fails to generalize to multivariate and multimodal time-series data.

Mukherjee, Subrata [ORNL] (ORCID:0000000309930338)↗

Metering Plan: Monitoring Energy and Potable Water Use in PNNL EMS4 Buildings [Slides]

This Plan presents progress toward the metering goals shared by all national laboratories and discusses PNNL's contemporary approach to the installation of new meters. In addition, the Plan discusses the data analysis techniques with which PNNL is working to mature using endless data streams made available as a result of increased meter deployment.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Slow control and TDAQ systems installation and tests in the Mu2e experiment

The Mu2e experiment at Fermilab will attempt to detect a coherent neutrinoless conversion of a muon into an electron in the field of an aluminum nucleus, with a sensitivity that is 10,000 times greater than existing limits. The Mu2e trigger and data acquisition system (TDAQ) uses the otsdaqframework as its online Data Acquisition System (DAQ) solution. Developed at Fermilab, otsdaq integrates several components, such as an artdaq-based DAQ, an art-based event processing, and an EPICS-based detector control system (DCS), and provides a uniform multi-user interface toits components through a web browser. The data streams from the Mu2e tracker and calorimeter are handled by the artdaq-based DAQ and processed by a one-level software trigger implemented within the art framework. Events accepted by the trigger have their data combined, post-trigger, with the separately read-out data from the Mu2e Cosmic Ray Veto system. The foundation of Mu2e DCS, EPICS, an Experimental Physics and Industrial Control System, is an open-source platform for monitoring, controlling, alarming, and archiving. Over the last three years, a prototype ofthe TDAQ and DCS systems has been built and tested at Fermilab’s Feynman Computing Center.Currently, the production system installation is underway. At the end, this work presents a brief update on the installation of racks and DAQ hardware.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Operational Focused Data Analytics for Optimizing Radiation Portal Monitor-Based Nuclear Smuggling Detection Systems at Global Ports of Entry

The National Nuclear Security Administration’s Office of Nuclear Smuggling Detection and Deterrence has deployed a fleet of radiation portal monitors (RPMs) across the world at global ports of entry including seaports, airports, and land border crossings. These RPMs are integrated into radiation detection systems (RDS) that also include fixed cameras, optical character recognition (OCR) systems, primary scanning systems (e.g., X-ray or gamma-ray), and secondary scanning systems (e.g., spectroscopic radiation portal monitors, portable radiation detection systems). The data from these sensing technologies is collected at the Central Alarm Station (CAS) where servers and computers reside to control and operate the system. Operators utilize the data collected by the CAS and declared cargo information to make decisions on how to respond to an alarm.This work explores the use of CAS-located data, looking at both the sensor data streams and operator inputs, to perform analysis which supports customs and border protection agencies to improve training capability and operational effectiveness. We focus on analyzing site level effectiveness and behavior by rolling up CAS-located data collected from individual occurrences. To-date, more than 15 sites (e.g., seaports, airports, border crossings) have been analyzed in this manner with the goal of understanding system operations to verify effectiveness and recommend potential improvements. This work first aims to provide background information on relevant CAS-located data sources and our current operational system analytics process including example results. After summarizing our current analytic techniques, we discuss how the future data analytics systems can provide key benefits to improving operational performance while minimizing the burden these detection systems place on operators.

Kuhn, Michael↗

Hysteresis Patterns of Watershed Nitrogen Retention and Loss Over the Past 50 years in United States Hydrological Basins

Patterns of watershed nitrogen (N) retention and loss are shaped by how watershed biogeochemical processes retain, biogeochemically transform, and lose incoming atmospheric deposition of N. Loss patterns represented by concentration, discharge, and their associated stream exports are important indicators of integrated watershed N retention behaviors. We examined continental United States (CONUS) scale N deposition (e.g., wet and dry atmospheric deposition), vegetation trends, and stream trends as potential indicators of watershed N-saturation and retention conditions, and how watershed N retention and losses vary over space and time. By synthesizing changes and modalities in watershed nitrogen loss patterns based on stream data from 2200 U.S. watersheds over a 50 years record, our work revealed two patterns of watershed N-retention and loss. One was a hysteresis pattern that reflects the integrated influence of hydrology, atmospheric inputs, land-use, stream temperature, elevation, and vegetation. The other pattern was a one-way transition to a new state. We found that regions with increasing atmospheric deposition and increasing vegetation health/biomass patterns have the highest N-retention capacity, become increasingly N-saturated over time, and are associated with the strongest declines in stream N exports—a pattern, that is, consistent across all land cover categories. We provide a conceptual model, validated at an unprecedented scale across the CONUS that links instream nitrogen signals to upstream mechanistic landscape processes. Our work can aid in the future interpretation of in-stream concentrations of DOC and DIN as indicators of watershed N-retention status and integrators of watershed hydrobiogeochemical processes.

58 GEOSCIENCES↗

Quillinan, et al 2018 DOE Geothermal Technology Office REE Report for NEWTS Database and Case Studies

Produced water data processed into the NEWTS data format for easy input into aqueous chemistry modeling software, including oil & gas and coal bed methane produced waters, and geothermal waters. Includes information on rare earth elements (REEs) and critical minerals (CMs). Case studies are included to demonstrate solved streams using aqueous chemistry software. An input template is provided for modeling stream data in OLI Studio. Original data from: Quillinan, Scott, Nye, Charles, Engle, Mark, Bartos, Timothy T., Neupane, Ghanashyam, Brant, Jonathan, Bagdonas, Davin, McLing, Travis, McLaughlin, J. Fred, Phillips, Erin, Hallberg, Laura L., Shahabadi, Mahdi, and Johnson, Matthew. Assessing rare earth element concentrations in geothermal and oil and gas produced waters: A potential domestic source of strategic mineral commodities (Final Report). United States: N. p., 2018. Web. doi:10.2172/1509037.

Aqueous Chemistry↗

An Accurate and Automated Convective Vortex Detection Method for Long-Duration Infrasound Microbarometer Data

Heating of the surficial layer of the atmosphere often generates convective vortices, known as “dust devils” when they entrain visible debris. Convective vortices are common on both Earth and Mars, where they affect the climate via dust loading, contribute to wind erosion, impact the efficiency of photovoltaic systems, and potentially result in injury and property damage. However, long-duration terrestrial convective vortex activity records are rare. We have developed a high-precision and high-recall method to extract convective vortex signatures from infrasound microbarometer data streams. The techniques utilizes a wavelet-based detector to capture potential events and then a template matching system to extract the duration of the vortex. Since permanent and temporary infrasound sensors networks are present throughout the globe (many with open data), our method unlocks a vast new convective vortex dataset without requiring the deployment of specialized instrumentation.

54 ENVIRONMENTAL SCIENCES↗

INITIAL STUDIES OF CAVITY FAULT PREDICTION AT JEFFERSON LABORATORY

The Continuous Electron Beam Accelerator Facility (CEBAF) at Jefferson Laboratory is a CW recirculating linac that utilizes over 400 superconducting radio-frequency (SRF) cavities to accelerate electrons up to 12 GeV through 5-passes. Recent work has shown that, given RF signals from a cavity during a fault as input, machine learning approaches can accurately classify the fault type. In this paper we report on initial results of predicting a fault onset using only data prior to the failure event. A data set was constructed using time-series data immediately before a fault (’unstable’) and 1.5 seconds prior to a fault (’stable’) gathered from over 5,000 saved fault events. The data was used to train a binary classifier. The results gave key insights into the behavior of several fault types and provided motivation to investigate whether data prior to a failure event could also predict the type of fault. We discuss our method using a sliding window approach and report on initial results. Recent modifications to the low-level RF control system will provide access to streaming signals and we outline a path forward for leveraging deep learning on streaming data

Vidyaratne, Lasitha↗