Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 757 records · Page 42

FPGA based demodulation of laser induced fluorescence in plasmas

We present a field programmable gate array (FPGA)-based system that counts photons from laser-induced fluorescence (LIF) on a laboratory plasma. This is accomplished with FPGA-based up/down counters that demodulate the data, giving a background-subtracted LIF signal stream that is updated with a new point as each laser amplitude modulation cycle completes. We demonstrate using the FPGA to modulate a laser at 1 MHz and demodulate the resulting LIF data stream. Finally, this data stream is used to calculate an LIF-based measurement sampled at 1 MHz of a plasma ion fluctuation spectrum.

47 OTHER INSTRUMENTATION↗

Note: Optical receiver system for 152-channel magnetoencephalography

An optical receiver system composing 13 serial data restore/synchronizer modules and a single module combiner converted optical 32-bit serial data into 32-bit synchronous parallel data for a computer to acquire 152-channel magnetoencephalography (MEG) signals. A serial data restore/synchronizer module identified 32-bit channel-voltage bits from 48-bit streaming serial data, and then consecutively reproduced 13 times of 32-bit serial data, acting in a synchronous clock. After selecting a single among 13 reproduced data in each module, a module combiner converted it into 32-bit parallel data, which were carried to 32-port digital input board in a computer. When the receiver system together with optical transmitters were applied to 152-channel superconducting quantum interference device sensors, this MEG system maintained a field noise level of 3 fT/√Hz @ 100 Hz at a sample rate of 1 kSample/s per channel.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

Performance of the Ultra-High Rate Germanium (UHRGe) System

This report describes the final performance achieved with the detector system developed for the Ultra High Rate Germanium (UHRGe) project. The system performance has been evaluated at low, moderate and high rates and includes the performance of real-time analysis algorithms running in the FPGA of the data acquisition system. This performance is compared to that of offline analyses of streaming waveform data collected with the same data acquisition system the performance of a commercial Multi-Channel Analyzer designed for high-resolution spectroscopy applications, the Canberra LYNX.

47 OTHER INSTRUMENTATION↗

Bio-energy feedstock yields and their water quality benefits in Mississippi

Cellulosic and agricultural bio-energy crops can, under careful management, be harvested as feedstock for bio-fuels production and provide environmental benefits. However, it is required to quantify their relative advantages in feedstock production and water quality. The primary objective of this research was to evaluate potential feedstock yield and water quality benefit scenarios of bioenergy crops: Miscanthus (Miscanthus-giganteus), Switchgrass (Panicum virgatum), Johnsongrass (Sorghum halepense), Alfalfa (Medicago sativa L.), Soybean {Glycine max (L.) Merr.}, and Corn (Lea mays) in the Upper Pearl River watershed (UPRW), Mississippi using a Soil and Water Assessment Tool (SWAT). The SWAT model was calibrated (January 1981 to December 1994) and validated (January 1995 to September 2008) using monthly measured stream flow data. The calibrated and validated model determined good to very good performance for stream flow prediction (R2 and E from 0.60 to 0.86). The RMSE values (from 14 m3 s-1 to 37 m3 s-1) were estimated at similar levels of errors during model calibration and validation. The long-term average annual potential feedstock yield as an alternative energy source was determined the greatest when growing Miscanthus grass (373,849 Mg) as followed by Alfalfa (206,077 Mg), Switchgrass (132,077 Mg), Johnsongrass (47,576 Mg), Soybean (37,814 Mg), and Corn (22,069 Mg) in the pastureland and cropland of the watershed. Model results determined that average annual sediment yield from the Miscanthus grass scenario determined the least (1.16 Mg/ha) and corn scenario the greatest (12.04 Mg/ha). The SWAT model simulated results suggested that growing Miscanthus grass in the UPRW would have the greatest potential feedstock yield and water quality benefits.

Biofuels↗

Towards adaptive, streaming analysis of x-ray tomography data

Temporal and spatial resolution of chemical imaging methodologies such as x-ray tomography are rapidly increasing, leading to more complex experimental procedures and fast growing data volumes. Automated analysis pipelines and big data analytics are becoming essential to effectively evaluate the results of such experiments. Offering those data techniques in an adaptive, streaming environment can further substantially improve the scientific discovery process, by enabling experimental control and steering based on the evaluation of emerging phenomena as they are observed by the experiment. Pacific Northwest National Laboratory (PNNL)’ Chemical Imaging Initiative (CII - http://imaging.pnnl.gov/ ) has worked since 2011 towards developing a framework that allows users to rapidly compose and customize high throughput experimental analysis pipelines for multiple instrument types. The framework, named ‘Rapid Experimental Analysis’ (REXAN) Framework [1], is based on the idea of reusable component libraries and utilizes the PNNL developed collaborative data management and analysis environment ‘Velo’, to provide a user friendly analysis and data management environment for experimental facilities. This article will, discuss the capabilities established for X-Ray tomography, discuss lessons learned, and provide an overview of our more recent work in the Analysis in Motion Initiative (AIM - http://aim.pnnl.gov/ ) at PNNL to provide REXAN capabilities in a streaming environment.

analysis framework↗

Random Forest Regressor-Based Approach for Detecting Fault Location and Duration in Power Systems

Power system failures or outages due to short-circuits or “faults” can result in long service interruptions leading to significant socio-economic consequences. It is critical for electrical utilities to quickly ascertain fault characteristics, including location, type, and duration, to reduce the service time of an outage. Existing fault detection mechanisms (relays and digital fault recorders) are slow to communicate the fault characteristics upstream to the substations and control centers for action to be taken quickly. Fortunately, due to availability of high-resolution phasor measurement units (PMUs), more event-driven solutions can be captured in real time. In this paper, we propose a data-driven approach for determining fault characteristics using samples of fault trajectories. A random forest regressor (RFR)-based model is used to detect real-time fault location and its duration simultaneously. This model is based on combining multiple uncorrelated trees with state-of-the-art boosting and aggregating techniques in order to obtain robust generalizations and greater accuracy without overfitting or underfitting. Four cases were studied to evaluate the performance of RFR: 1. Detecting fault location (case 1), 2. Predicting fault duration (case 2), 3. Handling missing data (case 3), and 4. Identifying fault location and length in a real-time streaming environment (case 4). A comparative analysis was conducted between the RFR algorithm and state-of-the-art models, including deep neural network, Hoeffding tree, neural network, support vector machine, decision tree, naive Bayesian, and K-nearest neighborhood. Experiments revealed that RFR consistently outperformed the other models in detection accuracy, prediction error, and processing time.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Multi-Level Anomaly Detection on Time-Varying Graph Data

This work presents a novel modeling and analysis framework for graph sequences which addresses the challenge of detecting and contextualizing anomalies in labelled, streaming graph data. We introduce a generalization of the BTER model of Seshadhri et al. by adding flexibility to community structure, and use this model to perform multi-scale graph anomaly detection. Specifically, probability models describing coarse subgraphs are built by aggregating probabilities at finer levels, and these closely related hierarchical models simultaneously detect deviations from expectation. This technique provides insight into a graph's structure and internal context that may shed light on a detected event. Additionally, this multi-scale analysis facilitates intuitive visualizations by allowing users to narrow focus from an anomalous graph to particular subgraphs or nodes causing the anomaly. For evaluation, two hierarchical anomaly detectors are tested against a baseline Gaussian method on a series of sampled graphs. We demonstrate that our graph statistics-based approach outperforms both a distribution-based detector and the baseline in a labeled setting with community structure, and it accurately detects anomalies in synthetic and real-world datasets at the node, subgraph, and graph levels. To illustrate the accessibility of information made possible via this technique, the anomaly detector and an associated interactive visualization tool are tested on NCAA football data, where teams and conferences that moved within the league are identified with perfect recall, and precision greater than 0.786.

anomaly detection↗

Streaming Readout of the CLAS12 Forward Tagger Using TriDAS and JANA2

An effort is underway to develop streaming readout data acquisition system for the CLAS12 detector in Jefferson Lab’s experimental Hall-B. Successful beam tests were performed in the spring and summer of 2020 using a 10GeV electron beam from Jefferson Lab’s CEBAF accelerator. The prototype system combined elements of the TriDAS and CODA data acquisition systems with the JANA2 analysis/reconstruction framework. This successfully merged components that included an FPGA stream source, a distributed hit processing system, and software plugins that allowed offline analysis written in C++ to be used for online event filtering. Details of the system design and performance are presented.

Ameli, Fabrizio↗

Automated System-wide Event Detection and Classification Using Machine Learning on Synchrophasor Data

As the number of phasor measurement units (PMUs) deployed in a power system increases, and their data volume streamed to the control canter intensifies, operators are facing challenges related to the analysis of such data, which need to be observed and responded to as the measurements are displayed in the Control Room. Humans are generally unable to process such large amount of data efficiently and rapidly. There is an apparent need for automated ways to analyze the data, extract actionable information about occurrence of specific events, and characterize the events quickly and cost effectively. This paper discusses the use of machine learning (ML) to facilitate such tasks by providing automated, highly computationally efficient, and cost-effective ways of extracting actionable information from synchrophasor big data in real-time. We developed Big Data Smart (BDSmart) ML-based prototype tool for the Control Room use that automatically analyses data properties from synchrophasor system measurements taken across the three grid Interconnections in the USA (Western, Eastern and ERCOT). The data collected from several hundreds of PMUs located across the Interconnections over a period of two years have been made available for our extensive study. As a result, we were able to identify a number of big data properties that influence how ML methodology is applied to select, develop, train and test the data models that can eventually be used for the tool implementation. The resulting set of candidate algorithms spans unsupervised, supervised, semi-supervised and transfer-learning approaches. Many ML techniques, such as decision trees, multinomial logistic regression, feed-forward neural networks, K-nearest neighbor, multiclass support vector machine, and single and multi-channel convolutional neural networks, are implemented, and their performance is examined. We offer the results from testing the data models. The novelty of our study is in the approaches for bad data detection and mitigation, selection of a simplified feature for event detection, and data label improvements. As a result, we came up with a list of recommendations for the utilities on how to improve the PMU recording practices to cater to the future ML applications aimed at automating the analysis of synchrophasor data.

Synchrophasors, Machine Learning, System-wide Even↗

A Novel Event Detection Method Using PMU Data with High Precision

To take full advantage of the considerably high reporting rate of phasor measurement units (PMU) data, this paper develops a novel PMU-based event detection methodology. Considering the huge amount of streaming PMU data, a data compression algorithm, swinging door trending (SDT), is first used to compress the PMU data and generate multiple compression intervals. Then dynamic programming is utilized to solve the optimization problem, which is recursively constituted by a score function. Based on predefined PMU event rules, dynamic programming merges adjacent compression intervals with the same slope direction. Finally, all the PMU event features are characterized. A conventional wavelet-based event detection method is compared with the developed dynamic programming based SDT (DPSDT) method. Numerical simulations on the real-time and synthetic PMU data show that the DPSDT method can accurately detect the start-time of an event and the event placement with relatively high precision. Furthermore, the PMU event features, including the magnitude and duration of strokes, are characterized.

24 POWER TRANSMISSION AND DISTRIBUTION↗

DELVE-ing into the Jet: A Thin Stellar Stream on a Retrograde Orbit at 30 kpc

We perform a detailed photometric and astrometric analysis of stars in the Jet stream using data from the first data release of the DECam Local Volume Exploration Survey DR1 and Gaia EDR3. We discover that the stream extends over ∼ 29° on the sky (increasing the known length by 18°), which is comparable to the kinematically cold Phoenix, ATLAS, and GD-1 streams. Using blue horizontal branch stars, we resolve a distance gradient along the Jet stream of 0.2 kpc deg$^{−1}$, with distances ranging from D $_{⊙}$ ∼ 27–34 kpc. We use natural splines to simultaneously fit the stream track, width, and intensity to quantitatively characterize density variations in the Jet stream, including a large gap, and identify substructure off the main track of the stream. Furthermore, we report the first measurement of the proper motion of the Jet stream and find that it is well aligned with the stream track, suggesting the stream has likely not been significantly perturbed perpendicular to the line of sight. Finally, we fit the stream with a dynamical model and find that it is on a retrograde orbit, and is well fit by a gravitational potential including the Milky Way and Large Magellanic Cloud. These results indicate the Jet stream is an excellent candidate for future studies with deeper photometry, astrometry, and spectroscopy to study the potential of the Milky Way and probe perturbations from baryonic and dark matter substructure.

79 ASTRONOMY AND ASTROPHYSICS↗

A multi-level anomaly detection algorithm for time-varying graph data with interactive visualization

This work presents a novel modeling and analysis framework for graph sequences which addresses the challenge of detecting and contextualizing anomalies in labelled, streaming graph data. We introduce a generalization of the BTER model of Seshadhri et al. by adding flexibility to community structure, and use this model to perform multi-scale graph anomaly detection. Specifically, probability models describing coarse subgraphs are built by aggregating node probabilities, and these related hierarchical models simultaneously detect deviations from expectation. This technique provides insight into a graph's structure and internal context that may shed light on a detected event. Additionally, this multi-scale analysis facilitates intuitive visualizations by allowing users to narrow focus from an anomalous graph to particular subgraphs or nodes causing the anomaly. For evaluation, two hierarchical anomaly detectors are tested against a baseline Gaussian method on a series of sampled graphs. We demonstrate that our graph statistics-based approach outperforms both a distribution-based detector and the baseline in a labeled setting with community structure, and it accurately detects anomalies in synthetic and real-world datasets at the node, subgraph, and graph levels. Furthermore, to illustrate the accessibility of information made possible via this technique, the anomaly detector and an associated interactive visualization tool are tested on NCAA football data, where teams and conferences that moved within the league are identified with perfect recall, and precision greater than 0.786.

97 MATHEMATICS AND COMPUTING↗

Analysis of material recovery facilities for use in life-cycle assessment

Highlights: • Life-cycle assessment of solid waste management relies on accurate process models. • Material recovery facility (MRF) processes were modeled with new primary data. • Single stream, dual stream, pre-sorted, and mixed waste MRFs were considered. • MRF electricity consumption ranges from 4.7 to 7.8 kW h per Mg input. • Total cost ranges from $19.8 to $24.9 per Mg input. - Abstract: Insights derived from life-cycle assessment of solid waste management strategies depend critically on assumptions, data, and modeling at the unit process level. Based on new primary data, a process model was developed to estimate the cost and energy use associated with material recovery facilities (MRFs), which are responsible for sorting recyclables into saleable streams and as such represent a key piece of recycling infrastructure. The model includes four modules, each with a different process flow, for separation of single-stream, dual-stream, pre-sorted recyclables, and mixed-waste. Each MRF type has a distinct combination of equipment and default input waste composition. Model results for total amortized costs from each MRF type ranged from $19.8 to $24.9 per Mg (1 Mg = 1 metric ton) of waste input. Electricity use ranged from 4.7 to 7.8 kW h per Mg of waste input. In a single-stream MRF, equipment required for glass separation consumes 28% of total facility electricity consumption, while all other pieces of material recovery equipment consume less than 10% of total electricity. The dual-stream and mixed-waste MRFs have similar electricity consumption to a single-stream MRF. Glass separation contributes a much larger fraction of electricity consumption in a pre-sorted MRF, due to lower overall facility electricity consumption. Parametric analysis revealed that reducing separation efficiency for each piece of equipment by 25% altered total facility electricity consumption by less than 4% in each case. When model results were compared with actual data for an existing single-stream MRF, the model estimated the facility’s electricity consumption within 2%. The results from this study can be integrated into LCAs of solid waste management with system boundaries that extend from the curb through final disposal.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Pre-commercial Demonstration of Direct Non-iterative State Estimator for Operational Use of Synchrophasor Management Systems

The objective of this project was to demonstrate functionality and performance of a Direct Non-iterative State Estimator (DNSE) integrated with NYPA’s Energy Management System (EMS) and with enhanced Real Time Dynamics Monitoring System (RTDMS) synchrophasor platform from Electric Power Group (EPG). DNSE is designed to overcome a major obstacle to operational use of Synchro-Phasor Management Systems (SPMS) by providing to synchrophasor management systems (SPMS) applications a consistent and a complete synchrophasor data foundation in the same way that a traditional EMS State Estimator (SE) provides to EMS applications. Specifically, DNSE is designed to use synchrophasor measurements collected by a central PDC, Supervisory Control and Data Acquisition (SCADA) measurements, and Energy Management System (EMS) network model, to obtain the complete state of the utility’s operating model at rates that are close to the synchrophasor data rates. In this way, the system is comprehensive in that it does not only cover the part of the network that is “visible” via synchrophasors, but also the part that is only “visible” through the SCADA measurements. Visualization needs associated with the use of DNSE results are fulfilled through suitably enhanced Real Time Dynamics Monitoring System (RTDMS), with the enhancements implemented by EPG. This project had the following goals in mind: To advance the deployment of commercial grade DNSE software application that relies on synchrophasor and SCADA data ; Apply DNSE at other utilities, to address a generic and fundamental need for “clean” operational data for synchrophasor applications; Provide means for “live” estimated data access by control system operators; Enhance potential for situational awareness through full system operational model coverage; Sub-second execution rate of the Direct Non-iterative State Estimator, eventually at a near-phasor data rate execution speed, i.e. < 0.1 sec. Anticipated benefits from this projects are: Enhanced reliability and improvements in the economic efficiency of bulk power system planning and operations; Providing “clean” data to other synchrophasor applications; Enhancement of situational awareness by providing the full operational model updated at near synchrophasor rate; A production-grade software tool that incorporate synchrophasor and SCADA data; Provides a basis for development of next generation monitoring and control applications, based on both SCADA and PMU data. Quanta Technology (QT) team worked in collaboration with Electric Power Group (EPG) whose team has enhanced its commercial Real Time Dynamics Monitoring System (RTDMS) to accommodate the requirements posed by DNSE application. EPG also provided its ePDC and Model-less Data Conditioning (PDVC) software for integration with DNSE+. QT developed the system requirements for DNSE; developed system architecture and defined interfaces between internal DNSE components. The core DNSE algorithm with all surrounding interfaces was named DNSE+. Since the DNSE development was done in a simulated system environment, QT used its PMU simulator that was enhanced during this project for development and factory acceptance testing (FAT). SCADA data in this stage was simulated by commercial PSS/e software. The output of DNSE are estimates of System states in C37.118-2 format, sent to RTDMS for further processing and display. As the number of these states is large, it was necessary to expand the C37.111-2 standard to accommodate large data sets. This enhancement was implemented in RTDMS. The demonstration of pre-production DNSE technology was done at NYPA using streaming field data from NYPA PMUs and from its RTUs through their SCADA system. NYPA provided ICCP interface as well as Common Information Model (CIM). The relevance of the DNSE+ application is that it provides state estimation of the power system based on hybrid set of data, consisting of both available PMU data and SCADA measurements. As this is a direct, non-iterative method of calculation of the system states, if does not suffer from convergence issues which is potential problem for conventional state estimators. Also, it can take any available PMU measurements, so it does not need to have a high percentage of PMU coverage needed in the case of Linear State Estimator. As the DNSE calculates synchrophasors of the system states (both phase and absolute value) as sub-second rate, this application can provide a basis for development of next generation of applications based both on SCADA and PMU data.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Hardware Architectures for Data-Intensive Computing Problems: A Case Study for String Matching

DNA analysis is an emerging application of high performance bioinformatic. Modern sequencing machinery are able to provide, in few hours, large input streams of data, which needs to be matched against exponentially growing databases of known fragments. The ability to recognize these patterns effectively and fastly may allow extending the scale and the reach of the investigations performed by biology scientists. Aho-Corasick is an exact, multiple pattern matching algorithm often at the base of this application. High performance systems are a promising platform to accelerate this algorithm, which is computationally intensive but also inherently parallel. Nowadays, high performance systems also include heterogeneous processing elements, such as Graphic Processing Units (GPUs), to further accelerate parallel algorithms. Unfortunately, the Aho-Corasick algorithm exhibits large performance variability, depending on the size of the input streams, on the number of patterns to search and on the number of matches, and poses significant challenges on current high performance software and hardware implementations. An adequate mapping of the algorithm on the target architecture, coping with the limit of the underlining hardware, is required to reach the desired high throughputs. In this paper, we discuss the implementation of the Aho-Corasick algorithm for GPU-accelerated high performance systems. We present an optimized implementation of Aho-Corasick for GPUs and discuss its tradeoffs on the Tesla T10 and he new Tesla T20 (codename Fermi) GPUs. We then integrate the optimized GPU code, respectively, in a MPI-based and in a pthreads-based load balancer to enable execution of the algorithm on clusters and large sharedmemory multiprocessors (SMPs) accelerated with multiple GPUs.

Tumeo, Antonino↗

System for detecting objects in streaming 3D images formed from data acquired with a medium penetrating sensor

A detection system that detects subsurface objects within a medium and estimates various features of the objects is provided. The detection system receives a streaming sequence of image frames of the medium at various along-medium locations. An image frame contains voxel values (intensities) representing characteristics of the medium across the medium and in the depth (range) direction. The detection system depth-compensates the intensities for determining which voxels are part of an object using an unsupervised binary classifier. The detection system then connects object voxels into distinct objects and recursively estimates the features of those objects as the image frames stream based on the locations and intensities of the object voxels.

Paglieroni, David W.↗

Data Federation Challenges in Remote Near-Real-Time Fusion Experiment Data Processing

Fusion energy experiments and simulations provide critical information needed to plan future fusion reactors. As next-generation devices like ITER move toward long-pulse experiments, analyses, including AI and ML, should be performed in a wide range of time and computing constraints, from near-real-time constraints, between-shot analysis, and to campaign-wide long-term analysis. However, the data volume, velocity, and variety make it extremely challenging for analyses using only local computational resources. Researchers need the ability to compose and execute workflows spanning edge resources to large-scale high-performance computing facilities.We present Delta, a system to address data analysis challenges, including AI/ML, in fusion science, by leveraging the ADIOS I/O library and middleware, to support executing science workflows over the wide area network for near-real-time streaming. We discuss the data federation challenges in performing remote workflows, focusing on on-going research work in (1) managing, reducing, and streaming data to minimize I/O and data movement overheads, (2) decompressing and reorganizing data for analysis, and (3) executing workflows for automated data analysis. We introduce examples for deep-learning based data analysis for the fusion domain and demonstrate how we use Delta to construct end-to-end workflows for a fusion device in Korea, connecting a remote DOE facility in the USA. The capability demonstrated by this project is the basis for improving the state of the art for near-real-time data federation amongst remote facilities.

Choi, Jong Youl↗