Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data streams”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17

Multimodal sensor fusion framework for residential building occupancy detection

For several years now, smart building energy systems have been a research area of intensive activity. In light of the increasing need for sustainable buildings and energy systems, this trend motivates an increasing need for a solution to reduce carbon dioxide emissions and improve energy efficiency. This work proposes a high-performing and transferable occupancy detection framework that combines sensor data from different data modalities, including time series environmental data (temperature, humidity, and illuminance), image data, and acoustic energy data using ensemble method. To draw out the best prediction performance in each modality, the proposed framework was developed, including various models that were designed to learn the occupancy patterns reflected in the physical data streams. To tackle the time series environmental data, we designed two variants of an occupancy detection spatiotemporal pattern network (Occ-STPN) that performs both feature level and decision level fusion, respectively. We also propose a new metric; the fading memory mean square error (FMMSE), that provides a fair evaluation and penalization of delayed occupancy predictions. Multiple open-sourced datasets, including the Electricity Consumption and Occupancy and the University of California, Irvine's (UCI) building occupancy detection dataset, along with our own real data collected from six different houses, were used to validate the algorithms' performance. The experimental results presented herein break down the performance for each sensing modality, and a detailed analysis of the performance is also discussed.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Nuclear data uncertainty propagation and modeling uncertainty impact evaluation in neutronics core simulation

Uncertainty analysis is a critical requirement in reactor simulation as it is used to quantify the reliability of best-estimate calculation. A comprehensive uncertainty analysis should characterize all sources of uncertainties in a computationally-feasible and scientifically-defendable manner. Here we employ a well-established reduced order modeling (ROM) based uncertainty quantification methodology to propagate uncertainties throughout neutronic calculations. ROM relies on recent advances in randomized data mining techniques applied to large data streams. In our proposed implementation, the nuclear data uncertainties are first propagated from multi-group level through lattice physics calculation to generate few-group parameter uncertainties, described using a vector of mean values and a covariance matrix. Employing an ROM-based compression of the covariance matrix, the few-group uncertainties are then propagated through downstream core simulation in a computationally efficient manner. This straightforward approach, albeit efficient as compared to brute force forward and/or adjoint-based methods, often employs a number of assumptions that have been unquestioned in the literature of neutronic uncertainty analysis. This manuscript argues that these assumptions could introduce another source of uncertainty referred to as modeling uncertainties, whose magnitude needs to be quantified in tandem with nuclear data uncertainties. Thus, our primary goal is to explore the interactions between these two uncertainty sources in order to assess whether modeling uncertainties have an impact on parameter uncertainties. To explore this endeavor, the impact of a number of modeling assumptions on core attributes uncertainties is quantified. The study employs a CANDU reactor model, with Serpent and NEWT as lattice physics solvers and NESTLE-C as core simulator. The modeling assumptions investigated include those related with the uncertainty propagation method employed, e.g., deterministic vs. stochastic, the few-group energy structure employed to represent the cross-sections, the resonance treatment in lattice physics calculation, the reference values for the cross-section, and the number of samples employed to render ROM compression. Results indicate that some of the modeling assumptions could have a non-negligible impact on the core responses propagated uncertainties, highlighting the need for a more comprehensive approach to combine parameter and modeling uncertainties.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

GNU Radio Sandia Utilities v. 1.6

This software adds a data handling module to the GNU Radio (GR) software defined radio (SDR) framework as well as some general-purpose function blocks (filters, metadata control, etc). This software is useful for processing bursty RF transmissions with GR, and serves as a base for applying SDR signal processing techniques to a whole burst of data at a time, as opposed to streaming data which GR.

Gilbert, Jacob↗

SmartBlock v. 1.0

SmartBlock offers reusable components that can provide data manipulation and transformation operations on different data streams without requiring recompilation for the new data types.

Champsaur, Alexis↗

Wattile: Probabilistic Deep Learning-based Forecasting of Building Energy Consumption [SWR-20-94]

Accurate energy forecasting is becoming critical due to many reasons: i ) optimal distributed energy resources operations and dispatch, ii) fault detection and diagnostics, and iii) meeting operational energy efficiency targets. Wattile uses deep learning (DL) for the building's short-term load forecasting application. Two specific types of neural networks called, Long Short Term Memory (LSTM) and Sequence-to-Sequence (S2S) models are used to make predictions. Forecasting models are trained using online historical weather and occupancy indicator data streams from the Intelligent Campus Program's data acquisition systems at the National Renewable Energy Laboratory (NREL) for main meters and sub-meters of multiple building types. These models use probabilistic methods to provide quantile-based forecasts in addition to nominal conditional median predictions of electricity consumption.

Frank, Stephen↗

AGR-2 Data Qualification Report for ATR Cycles 149B, 150A, 150B, 151A, and 151B

This report provides the data qualification status of AGR-2 fuel irradiation experimental data from Advanced Test Reactor (ATR) cycles 149B, 150A, 150B, 151A, and 151B), as recorded in the Nuclear Data Management and Analysis System (NDMAS). The AGR-2 data streams addressed include thermocouple temperatures, sweep gas data (flow rate, pressure, and moisture content), and fission product monitoring system (FPMS) data for each of the six capsules in the experiment. A total of 3,307,500 5-minute thermocouple and sweep gas data records were received and processed by NDMAS for this period. There are no AGR-2 data for cycle 150A because the experiment was removed from the reactor. Of these data, 82.2% were determined to be Qualified based on NDMAS accuracy testing and data validity assessment. There were 450,557 Failed temperature records due to thermocouple failures, and 138,528 Failed gas flow records due to gas flow cross-talk and leakage problems that occurred in the capsules after cycle 150A. For FPMS data, NDMAS received and processed preliminary release rate and release-to-birth rate ratio (R/B) data for the first three reactor cycles (cycles 149B, 150B, and 151B). This data consists of 45,983 release rate records and 45,235 R/B records for the 12 radionuclides reported. The qualification status of these FPMS data has been set to In Process until receipt of QA-approved data generator reports. All of the above data have been processed and tested using a SAS®-based enterprise application software system, stored in a secure Structured Query Language database, and made available on the NDMAS Web portal (http://ndmas.inl.gov) for both internal and external VHTR project participants.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

An Overview of ARM Program Climate Research Facility Data Quality Assurance

This report documents key aspects of the Atmospheric Radiation Measurement (ARM) Program's data quality assurance program. Processes described include data archival, display, and distribution, data stream reprocessing, and future directions in ARM data quality checking. A comprehensive, end-to-end data quality assurance program, from instrument siting, to calibration and maintenance, through data quality control and well-documented dissemination, is essential for producing a high-quality research data set. The processes developed over the past 15 years by the ARM Program offer a possible framework for use by other instrumentation-diverse networks, and are offered here to highlight the myriad aspects that go into producing such a data set.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Data fusion enhanced multi-modality wellbore integrity inspection system

A downhole multi-modality inspection system includes a first imaging device operable to generate first imaging data and a second imaging device operable to generate second imaging data. The first imaging device includes a first source operable to emit energy of a first modality, and a first detector operable to detect returning energy induced by the emitted energy of the first modality. The second imaging device includes a second source operable to emit energy of a second modality, and a second detector operable to detect returning energy induced by the emitted energy of the second modality. The system further includes a processor configured to receive the first imaging data and the second imaging data, and integrate the first imaging data with the second imaging data into an enhanced data stream. The processor correlates the first imaging data and the second imaging data to provide enhanced data for detecting potential wellbore anomalies.

Kasten, Ansas Matthias↗

Data Capture Technique for High Speed Signaling

A data capture technique for high speed signaling to allow for optimal sampling of an asynchronous data stream. This technique allows for extremely high data rates and does not require that a clock be sent with the data as is done in source synchronous systems. The present invention also provides a hardware mechanism for automatically adjusting transmission delays for optimal two-bit simultaneous bi-directional (SiBiDi) signaling.

Barrett, Wayne Melvin↗

Streaming Analytics and Data Engineering

SADE is a software package for rapidly assembling analytic pipelines to manipulate data. The packages consists of the engine that manages the data and coordinates the movement of data between the tasks performing a function, a set of core libraries consisting of plugins that perform common tasks, and a framework to extend the system supporting the development of new plugins. Currently through configuration files, a pipeline can be defined that maps the routing of data through a series of plugins. Pipelines can be run in a batch mode or can process streaming data; they can be executed from the command line or run through a Windows background service. There currently exists over a hundred plugins, over fifty pipeline configurations, and the software is now being used by about a half-dozen projects.

Source record↗

FTICR-MS, Sensor, and Environmental Data from 5 Streams Impacted by the 2020 Holiday Farm Fire Associated with: "Spatiotemporal controls on the delivery of dissolved organic matter to streams following a wildfire"

This data package is associated with the publication "Spatiotemporal Controls on the Delivery of Dissolved Organic Matter to Streams Following a Wildfire" submitted to Geophysical Research Letters (Roebuck et al., 2022). The study aims to understand storm induced transport of pyrogenic materials to streams impacted by varying degrees of burn severity. Time series samples (24 samples in 1-hour intervals) were collected at 5 sites within the McKenzie River Watershed (Oregon, USA) whose catchment were each completely engulfed by the 2020 Holiday Farm Fire. The samples were collected in November 2020 during the first major storm pulse following the conclusion of the wildfire. Samples were characterized for dissolved organic carbon, total dissolved nitrogen, and by ultra-high resolution mass spectrometry. In situ turbidity data also collected.This data package contains 4 primary folders that include the following: 1) Metadata, 2) EnvData (Environmental Data), 3) SensorData, and 4) FTICR_SupportingData. The package contains a single file-level metadata (flmd) file. Each primary folder also contains individual data dictionaries (dd) to define and provide descriptors of column/row headers and data flags. The FTICR_SupportingData, folder 4, contains raw, unprocessed FTICR-MS Data files in addition to a csv containing processed FTICR-MS data. This package contains the following file types: csv, xml, pdf.

54 ENVIRONMENTAL SCIENCES↗

Real-time data processing for serial crystallography experiments

We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames ('hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.

47 OTHER INSTRUMENTATION↗

Evaluating lightweight unsupervised online IDS for masquerade attacks in CAN

Vehicular controller area networks (CANs) are susceptible to masquerade attacks by malicious adversaries. In masquerade attacks, adversaries silence a targeted ID and then send malicious frames with forged content at the expected timing of benign frames. As masquerade attacks could seriously harm vehicle functionality and are the stealthiest attacks to detect in CAN, recent work has devoted attention to compare frameworks for detecting masquerade attacks in CAN. However, most existing works report offline evaluations using CAN logs already collected using simulations that do not comply with the domain’s real-time constraints. Here we contribute to advance the state of the art by presenting a comparative evaluation of four different non-deep learning (DL)-based unsupervised online intrusion detection systems (IDS) for masquerade attacks in CAN. Our approach differs from existing comparative evaluations in that we analyze the effect of controlling streaming data conditions in a sliding window setting. In doing so, we use realistic masquerade attacks being replayed from the ROAD dataset. We show that although evaluated IDS are not effective at detecting every attack type, the method that relies on detecting changes in the hierarchical structure of clusters of time series produces the best results at the expense of higher computational overhead. We discuss limitations, open challenges, and how the evaluated methods can be used for practical unsupervised online CAN IDS for masquerade attacks.

Anomaly detection↗

Efficient Modeling of HIL Multi-Grid System for Scalability & Concurrency in CPS Security Testbed

Cyber-event-triggered power grid blackout compels utility operators to intensify cyber-aware and physics-constrained recovery and restoration process. Recently, coordinated cyber attacks on the Ukrainian grid witnessed such a cyber-event-triggered power system blackout. Various cyber-physical system (CPS) testbeds have attempted with multitude designs to analyze such interdependent events and evaluate remedy measures. However, resource constraints and modular integration designs have been significant barriers while modeling large-scale grid models (scalability) and multi-grid isolated models (concurrency) under a single real-time execution environment for the hardware-in-the-loop (HIL) CPS security testbeds. This paper proposes a meticulous design and effective modeling for simulating large-scale grid models and multi-grid isolated models in a HIL realtime digital simulator environment integrated with industry-grade hardware and software systems. We have used our existing HIL CPS security testbed to demonstrate scalability by the realtime performance of a Texas-2000 bus US synthetic grid model and concurrency by the real-time performance of simultaneous ten IEEE-39 bus grid models and an IEEE-118 bus grid model. The experiments demonstrated significant results by 100% realtime performance with zero overruns, low latency while receiving and executing control signals from SEL Relays via IEC-61850 protocol and low latency while computing and transmitting grid data streams including stability measures via IEEE C37.118 synchrophasor data protocol to SEL Phasor Data Concentrators.

24 POWER TRANSMISSION AND DISTRIBUTION↗

AGR-2 Data Qualification Report for ATR Cycles 151B-2, 152A, 152B, 153A, 153B and 154A

This report documents the data qualification status of AGR-2 fuel irradiation experimental data from Advanced Test Reactor (ATR) Cycles 152A, 152B, 153A, 153B, and 154A, as recorded in the Nuclear Data Management and Analysis System (NDMAS). The AGR-2 data streams addressed include thermocouple (TC) temperatures, sweep gas data (flow rate, pressure, and moisture content), and fission product monitoring system (FPMS) data for each of the six capsules in the experiment. A total of 13,400,520 every minute instantaneous TC and sweep gas data records were received and processed by NDMAS for this period. Of these data, 8,911,791 records (66.5% of the total) were determined to be Qualified based on NDMAS accuracy testing and data validity assessment. For temperature, there were 4,266,081 records (74% of the total TC data) that were Failed due to TC instrument failures. For sweep gas flows, there were 222,648 gas flow records (2.91% of the flow data) that were Failed. The inlet gas flow failures due to gas flow cross-talk and leakage problems that occurred after Cycle 150A were corrected by using the same gas mixture in all six capsules and the Leadout. For FPMS data, NDMAS received and processed preliminary release rate and release-to-birth rate ratio (R/B) data for three reactor cycles (Cycles 149B, 150B, and 151A) . This data consists of 45,983 release rate records and 45,235 R/B records for the 12 radionuclides reported. The qualification status of these FPMS data has been set to In Process until receipt of Quality Assurance-approved data generator reports. All of the above data have been processed and tested using a SAS®-based enterprise application software system, stored in a secure Structured Query Language database, made available on the NDMAS Web portal (http://ndmas.inl.gov), and approved by the INL STIM for release to both internal and appropriate external Very High Temperature Reactor Program participants.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

HAIMOS Ensemble Forecasts for Intra-day and Day- Ahead GHI, DNI and Ramps

The objective of this research is to develop a hybrid physics-based/data-driven forecast model to improve direct normal and global horizontal irradiance (DNI and GHI) prediction for horizons ranging from 1 to 72 hours. Project objectives also address key gaps in state-of-the-art solar forecasting: accurate probabilistic solar forecasts and the forecasting of large irradiance ramps (ramp onset and magnitude). The proposed model ensembles Numerical Weather Prediction (NWP) forecasts, determinist physics-based algorithms, and new-generation cloud cover products (high-resolution rapid refresh satellite images and Large Eddy Simulations). The result is the Hybrid Adaptive Input Model Objective Selection (HAIMOS) ensemble model. HAIMOS blends state of the art machine learning methodologies with physics-based models for cloud cover and cloud optical depth forecasts. The technical activities followed a two-pronged strategy. First, the preprocessing of data, the selection of inputs to the nonlinear approximators, the type of approximator and objective functions, and post-processing ensembling techniques included in HAIMOS were all optimized adaptively to find the best model for a specific goal (reduce DNI/GHI forecast error, improve the prediction of ramp onset, etc.). Second, a large effort was put in improving cloud identification and the forecast of cloud cover and cloud optical depth. To this end, new-generation cloud parametrization products were developed in this work. These include improved algorithms to assist in cloud identification, cloud classification and cloud parametrization from satellite images – three key factors in the accuracy of 1 to 6-hours irradiance forecasts and prediction of ramp onset. Furthermore, we also included cloud information extracted from high resolution rapid refresh satellite images (GOES-16) and Large Eddy Simulations (LES). LES was used to model the atmosphere in detail over locations of interest and produce cloud optical depth forecasts. Once these data streams were validated, they were used as input data to the HAIMOS forecast. The model was developed using data from several climatologically distinct locations with potential for high solar penetration. In the last year of the project, we conducted a validation campaign according to the guidelines stipulated by the Topic Area 1 project as described in the FOA. This effort brings, for the first time, proven machine-learning methodologies for generating state-of-the-art solar forecasts interweaved with detailed physics-based models for cloud detection, and cloud optical depth forecasts. HAIMOS will generate accurate irradiance probabilistic forecast to assist in reducing solar generation prediction error. Globally optimized solar forecast models are more likely to impact solar energy stakeholders. The goal of this project was to increase the state-of-the-art forecast skill from their present values of 10 to 35%. At the end of the project, we achieved between 30% and 50% forecast skill across a wide range of horizons for both GHI and DNI.

14 SOLAR ENERGY↗

A Parallel Graph Environment for Real-World Data Analytics Workflows

Economic competitiveness and national security depend increasingly on the insightful analysis of large data sets. The diversity of real-world data sources and analytic workflows impose challenging hardware and software requirements for parallel graph platforms. The irregular nature of graph methods is not supported well by the deep memory hierarchies of conventional distributed systems, requiring new processor and runtime system designs to tolerate memory and synchronization latencies. Moreover, the efficiency of relational table operations and matrix computations are not attainable when data is stored in common graph data structures. In this paper, we present HAGGLE, a high-performance, scalable data analytics platform. The platform’s hybrid data model supports a variety of distributed, thread-safe data structures, parallel programming constructs, and persistent and streaming data. An abstract runtime layer enables us to map the stack to conventional, distributed computer systems with accelerators. The runtime uses multithreading, active messages, and data aggregation to hide memory and synchronization latencies on large-scale systems.

Graph Analytics↗

Managing high-bandwidth real-time data storage

There exist certain systems which generate real-time data at high bandwidth, but do not necessarily require the long-term retention of that data in normal conditions. In some cases, the data may not actually be useful, and in others, there may be too much data to permanently retain in long-term storage whether it is useful or not. However, certain portions of the data may be identified as being vitally important from time to time, and must therefore be retained for further analysis or permanent storage without interrupting the ongoing collection of new data. We have developed a system, Mahanaxar, intended to address this problem. It provides quality of service guarantees for incoming real-time data streams and simultaneous access to already-recorded data on a best-effort basis utilizing any spare bandwidth. It has built in mechanisms for reliability and indexing, can scale upwards to meet increasing bandwidth requirements, and handles both small and large data elements equally well. We will show that a prototype version of this system provides better performance than a flat file (traditional filesystem) based version, particularly with regard to quality of service guarantees and hard real-time requirements.

99 GENERAL AND MISCELLANEOUS↗