Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,171 records · Page 65

Developing a Decision Support Engine to Enable Irrigation Modernization - Poster

Irrigation systems in the United States are some of the oldest continually utilized infrastructure in existence today, with some systems exceeding 100 years in age. They are operated to meet farming demands but are managed through a balance of varying influence: policies limiting water usage, stakeholder interests, and environmental impacts. Irrigation modernization is defined as a set of activities that update and improve existing irrigation systems, including, but not limited to improving water quantity, development of distributed energy resources for surrounding communities, ecosystem services, and improved agricultural yields. Modernizing an existing irrigation system can enable stakeholders to combat changing environmental and population demands but is difficult because the complexity involved in determining the potential benefits and consequences of irrigation modernization is high. We are combining a large amount of various geospatial, tabular, and temporal data with subject matter expertise into a decision support engine that will enable stakeholders to determine the benefits and consequences of irrigation modernization in their irrigation systems. A web-based GIS will allow the user to construct the modifications out of a palette of modernization options, which will be sent to the analytics engine for computations, and back to the web client for a graphical display and comparison of relevant metrics. Our development process involves four phases: 1) identify mechanisms of modernization, 2) identify data requirements, data streams, first principles and applicable algorithms necessary to quantify modernization mechanisms, 3) create ‘modules’ for each modernization mechanism, these modules will form the decision support engine, each capable of performing independently but can also inform other modules when needed, 4) Merge the decision support engine with a user interface, capable of ingesting user inputs and returning insights into the impacts of a modernization project as they relate to economic, environmental, monetary, and energy generation. Once complete, it is our intention that this tool will be fundamental in irrigation modernization projects, providing a strong analytical basis from which stakeholders can quickly make informed decisions regarding project development.

13 HYDRO ENERGY↗

FPGA Architectures for Distributed ML Systems for Real-time Beam Loss De-blending

The Accelerator Real-time Edge AI for Distributed Systems (READS) project’s goal is to create a Artificial Intelligence (AI) system for real-time beam loss de-blending within the accelerator enclosure, which houses two accelerators: the Main Injector (MI) and the Recycler Ring (RR). In periods of joint operation, when both machines contain high intensity beam, radioactive beam losses from MI and RR overlap on the enclosure’s beam loss monitoring Beam Loss Monitor (BLM) system, making it difficult to attribute those losses to a single machine. Incorrect diagnoses result in unnecessary downtime that incurs both financialand experimental cost. The ML system will automatically disentangle each machine’s contributions to those measured losses, while not disrupting the existing operations-critical functions of the BLM system. This paper will focus on the evolution of the architectures, which provided the high-frequency, low-latency collection of synchronized data streams to make real-time inferences. The ML models, used for learning both local and global machine signatures and producing high quality inferences based on raw BLM loss measurements, will only be discussed at a high-level.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Single Carrier Modulation Resource Allocation for Massive MIMO with Virtual Antennas

Today’s Massive MIMO cellular operation is dominated by orthogonal frequency division multiplexing (OFDM) modulation. One of the advantages of OFDM is the flexibility to carve up the available spectrum into resource blocks (RBs) that can operate adjacent to one another. Massive MIMO adds spatial multiplexing layers on top of the RBs, enabling the simultaneous operation of dozens of UEs. Cyclic prefixed single carrier modulation (CP-SCM) is an alternative to OFDM that also benefits from massive MIMO and is useful as an OFDM alternative in specific scenarios. However, until recently, no resource allocation method has been proposed for CP-SCM. In this paper, we present a CP-SCM resource allocation scheme using virtual antennas in a massive MIMO time domain duplexed scenario. By creating several virtual antennas for each physical antenna, the benefits of massive MIMO are achieved with a smaller physical antenna count. Resources are quantized into data streams, and each user can be assigned a variable number of simultaneous streams. This paper presents a detailed development and analysis of multi-stream processing (MSP) for both uplink detection and downlink precoding. We also introduce heterogeneous MSP, where CP-SCM and OFDM signals can be processed in the same MSP framework.

5G and beyond communications↗

A Multi-Dimensional Benefit Assessment of Automated Mobility Platforms (AMP) for Large Facilities

Automated Mobility Platform (AMP) is a general term for intelligently managed passenger mobility systems that leverage automation and electrification to improve system performance and sustainability through the integration of real-time data streams with advanced decision-making processes tailored for use in large facilities. Using airports as an example, the rapid growth in air travel demand has led to facility expansions and congested terminals, which directly impacts equity (e.g., increased challenges for passengers with reduced mobility [PRMs]) and sustainability - both of which are important metrics often neglected during the engineering design process. Therefore, to evaluate systems and inform critical decisions more effectively, a holistic evaluation framework is proposed that includes equity, sustainability, and infrastructure perspectives - referred to as a multidimensional benefit assessment. More specifically, the following analysis focuses on: (1) mobility, with emphasis on travel time and accessibility within an airport, contrasting existing systems with the proposed AMP system; (2) equity analysis, specifically to the PRM community, but with an eye to the whole of society; (3) energy consumption and greenhouse gas (GHG) emissions associated with intra-airport mobility, comparing and contrasting modes as well as impacts on overall airport operations efficiency; and (4) fundamental changes in building design enabled by AMPs for larger and more flexible, functional, and energy-efficient structures.

ADVANCED PROPULSION SYSTEMS↗

RISMC Analysis Framework in Response to Proposed 10 CFR 50.46c Rulemaking

The results obtained from the demonstration calculations indicate that as an automation tool in LOCA analyses, LOTUS has the capability to provide integrated evaluation through seamless interfaces of the data stream in multiple disciplines. With further more development, LOTUS will provide a systematic way to perform uncertainty quantification, sensitivity analysis and risk assessment of the multiple physics involved in the LOCA analysis in response to the proposed 10 CFR 50.46c rulemaking. It can be used to obtain quantitative estimate of design or operational margin loss or gain as well as to assess the impact on LOCA analysis safety metrics associated with various combinations of changes in LOCA analysis inputs. It can also aid plant owners to make decision on evaluating design changes as well as resolving operational issues. (authors)

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Improved Line Outage Detection in Transmission Systems with Few PMUs

Unlike transmission systems, distribution systems historically lack enough measurements, making their real-time monitoring almost impossible. Recent deployment of diverse types of devices such as phasor measurement units (PMUs), smart meters, solar inverters and weather information sensors opens up new ways of monitoring these systems, with the assistance of customized machine learning (ML) applications. The paper describes a grid-model-informed machine learning (ML) tool which integrates heterogeneous data streams and creates synchronous measurement snapshots to be used by a hybrid robust state estimator (SE) which provides not only accurate state estimates but also real-time feedback for ML model refinement. Improved monitoring performance due to the use of developed computational framework is experimentally observed by simulated scenarios on an electric utility’s distribution system.

Distribution systems, graph learning, machine lear↗

The Sensor Dilemma in Intelligent Transportation Systems: Evaluating Radar, Lidar and Camera: Preprint

Intelligent transportation systems (ITS) are at the forefront in advancing the way we interact with and perceive the transportation network. This revolution is fueled by the significant advancement in sensor perception technologies such as radar, lidar, and video imaging, which are the most popular modalities for ITS. Real-time perception data from these sensors allow intelligent infrastructure-side decision-making to improve the energy, efficiency, and safety at traffic intersections. As traffic departments across the United States transition from traditional loop detectors and emulators and embrace newer technologies, they are often left with a dilemma in choosing a sensor technology for infrastructure-based perception that is reliable, inexpensive, and easy to set up and that has robust performance in varying weather conditions. However, choosing a sensor that checks all these boxes is not straightforward, as every sensor type has unique benefits and drawbacks. Radar is excellent at detecting long-range vehicles and weather resistance but lacks high resolution. Lidar is expensive and weather-sensitive, while cameras provide rich visual data at a low cost but are constrained by lighting and visibility. This study examines radar, lidar, and camera sensor capabilities to ascertain whether any of these qualifies as the "best" sensor for ITS perception. While no single sensor can meet all the demands of ITS, a hybrid approach combining multiple sensor modalities like radar, lidar, and cameras offers the most robust solution for enhancing the safety and efficiency of ITS. Through this evaluation, we hope to draw attention to the necessity of the National Renewable Energy Laboratory's infrastructure perception and control framework, which presents a multisensor track data fusion engine to assimilate multiple data streams in order to provide robust and reliable perception.

33 ADVANCED PROPULSION SYSTEMS↗

Neutrino beam bunch structure reconstruction with precision timing in the ICARUS liquid argon time projection chamber

The ICARUS detector has been operating smoothly since 2021 as the far detector in the Short Baseline Neutrino (SBN) program at Fermilab, collecting neutrino interactions from both the Booster Neutrino Beam (BNB) and off-axis from the Neutrinos at the Main Injector (NuMI) beam. Analysis of neutrino interactions in ICARUS requires mitigation of substantial cosmogenic backgrounds. This is achieved by using an external Cosmic Ray Tagger (CRT) and a Photomultiplier Tube (PMT) system embedded in the liquid argon. The intrinsic neutrino beam bunch structure, inherited time structure from the Radio Frequency (RF) system used to accelerate the protons, can be resolved at the ICARUS detector using precise timing information. Located at shallow depth, ICARUS is exposed to a high flux of cosmic rays that can be mistaken for neutrino interactions. To mitigate this background, the CRT and a 3-meter-thick concrete overburden were installed. To better model backgrounds, ICARUS makes use of an overlay technique where simulated neutrino events are superimposed on detector beam-off data. PMTs installed within a Time Projection Chamber detect argon scintillation light emitted by high energy charged particles passing through the chamber and provide the event timing of neutrino interactions. The nanosecond-level beam bunch structure is reconstructed with the PMT system and can be used to further understand backgrounds and enhance neutrino physics capabilities. In this thesis, I will discuss background mitigation techniques using precision timing and present a novel technique to select neutrino events from our unbiased data stream using the beam bunch structure.

Heggestuen, Anna [Colorado State U.] (ORCID:000000↗

EXCLUSIVE NEUTRAL PION ELECTROPRODUCTION CROSS SECTION MEASUREMENTSWITHANEUTRALPARTICLE SPECTROMETER

Deep Virtual Compton Scattering (DVCS), the exclusive electron-proton scattering process ep ¿e'p'¿, provides access to generalized parton distributions (GPDs), which correlate information about the longitudinal momentum and transverse spatial structure of quarks inside the nucleon. Experiment E12-13-010 in Hall C at Jefferson Lab was designed to take high-precision measurements of the DVCS cross section over an extended kinematic range using the newly commissioned Neutral Particle Spectrometer (NPS). The NPS features a high-resolution electromagnetic calorimeter and a streaming data acquisition system optimized for operation at high luminosities. This thesis presents the detector and analysis work carried out to support the NPS DVCS program. In particular, it focuses on the hardware design, calibration, and performance of the calorimeter. A development of a waveform reconstruction analysis of the calorimeter signals enabled improved extraction of pulse amplitudes and times. The waveform analysis was also extended to operate in a multithreaded environment, substantially reducing processing time for large datasets. Analysis of exclusive neutral pion electroproduction events in the calorimeter gives a strong validation of the calorimeter’s performance and resolution. Together these developments establish a foundation for future analyses and extraction of the DVCS cross section and its use in constraining the GPDs.

Kerver, Mitchell [Old Dominion Univ., Norfolk, VA ↗

Data from: “Significant stream chemistry response to temperature variations in a high-elevation mountain watershed”

High-elevation mountain regions, central to global freshwater supply, are experiencing more rapid warming than low-elevation locations. High-elevation streams are therefore potentially critical indicators for earth system and water chemistry response to warming. Here we present concerted hydroclimatic and biogeochemical data from Coal Creek, Colorado in the central Rocky Mountains at elevations of 2700 to 3700 m, where air temperatures have increased by about 2 °C since 1980. We analyzed water chemistry every other day from 2016 to 2019. Water chemistry data indicate distinct responses of different solutes to inter-annual hydroclimatic variations. Specifically, the concentrations of solutes from rock weathering are stable inter-annually. Solutes that are active in soils, including dissolved organic carbon, vary dramatically, with double to triple peak concentrations occurring during snowmelt and in warm years. We advocate for consistent and persistent monitoring of high elevation streams to record early glimpse of earth surface response to warming.1) Coal Creek stream were sampled and measured to answer the question if there is a significant stream chemistry response to climate change in high-elevation watershed2) The file contains data regarding discharge, water temperature and stream chemistry data for 2016-2019 year. Stream chemistry includes dissolved organic and inorganic carbon, chloride, nutrients, cations. The data collection efforts were carried out by the DOE-funded "Advancing a Watershed Hydro-Biogeochemical Theory: Linking Water Travel Time and Reaction Rates Under Changing Climate" project and Watershed Function SFA.

54 ENVIRONMENTAL SCIENCES↗

Investigating the impacts of solid phase extraction on dissolved organic matter optical signatures and the pairing with high‐resolution mass spectrometry data across a freshwater stream network

Abstract Advancing our understanding of dissolved organic matter (DOM) chemistry in aquatic systems necessitates the integration of data streams from multiple analytical platforms. Some measurements require pretreatment with solid phase extraction (SPE), while others are performed directly on whole water samples. Evidence has suggested that SPE will be biased against select DOM fractions, leading to concerns over the ability to establish data linkages across platforms with variable needs for SPE pretreatment, such as those from optical measurements and those that provide high‐resolution molecular information. Here, we directly addressed this concern by assessing the impact of SPE on DOM optical properties through excitation–emission matrices with parallel factor analysis (PARAFAC) for 47 samples across a stream network within a single watershed reflective of variable DOM sources. PARAFAC data was further paired with molecular information obtained by Fourier transform ion cyclotron resonance mass spectrometry (FTICR‐MS). A comparison of PARAFAC models first revealed no systematic qualitative differences in major components between whole water DOM and DOM isolated by SPE (SPE‐DOM); however, quantitative biases against select components were observed. Further linkages with FTICR‐MS data revealed that the molecular fingerprint associated with each PARAFAC component was consistent between the whole water DOM and SPE‐DOM. Our results suggest that bulk scale linkages across these analytical platforms could be inferred irrespective of the observed quantitative biases resulting from SPE for samples within this example watershed. This work represents a key step toward the systematic evaluation of linkages between optical and high‐resolution mass spectrometry datasets in freshwater lotic environments.

59 BASIC BIOLOGICAL SCIENCES↗

Large Scale Caching and Streaming of Training Data for Online Deep Learning

The training of deep neural network models on large data remains a difficult problem, despite progress towards scalable techniques. In particular, there is a mismatch between the random but predetermined order in which AI flows select training samples and the streaming I/O patterns for which traditional HPC data storage (e.g., parallel file systems) are designed. In addition, as more data are obtained, it is feasible neither simply to train learning models incrementally, due to catastrophic forgetting (i.e., bias towards new samples), nor to train frequently from scratch, due to prohibitive time and/or resource constraints. In this paper, we study data management techniques that combine caching and streaming with rehearsal support in order to enable efficient access to training samples in both offline training and continual learning. We revisit state-of-art streaming approaches based on data pipelines that transparently handle prefetching, caching, shuffling, and data augmentation, and discuss the challenges and opportunities that arise when combining these methods with data-parallel training techniques. We also report on preliminary experiments that evaluate the I/O overheads involved in accessing the training samples from a parallel file system (PFS) under several concurrency scenarios, highlighting the impact of the PFS on the design of the data pipelines.

data pipelines↗

Online anomaly detection for multi‐source VMware using a distributed streaming framework

Summary Anomaly detection refers to the identification of patterns in a dataset that do not conform to expected patterns. Such non‐conformant patterns typically correspond to samples of interest and are assigned to different labels in different domains, such as outliers, anomalies, exceptions, and malware. A daunting challenge is to detect anomalies in rapid voluminous streams of data. This paper presents a novel, generic real‐time distributed anomaly detection framework for multi‐source stream data. As a case study, we investigate anomaly detection for a multi‐source VMware‐based cloud data center, which maintains a large number of virtual machines (VMs). This framework continuously monitors VMware performance stream data related to CPU statistics (e.g., load and usage). It collects data simultaneously from all of the VMs connected to the network and notifies the resource manager to reschedule its CPU resources dynamically when it identifies any abnormal behavior from its collected data. A semi‐supervised clustering technique is used to build a model from benign training data only. During testing, if a data instance deviates significantly from the model, then it is flagged as an anomaly. Effective anomaly detection in this case demands a distributed framework with high throughput and low latency. Distributed streaming frameworks like Apache Storm, Apache Spark, S4, and others are designed for a lower data processing time and a higher throughput than standard centralized frameworks. We have experimentally compared the average processing latency of a tuple during clustering and prediction in both Spark and Storm and demonstrated that Spark processes a tuple much quicker than storm on average. Copyright © 2016 John Wiley & Sons, Ltd.

Solaimani, Mohiuddin↗

Statistical Techniques For Real-time Anomaly Detection Using Spark Over Multi-source VMware Performance Data

Anomaly detection refers to the identi cation of an irregular or unusual pat- tern which deviates from what is standard, normal, or expected. Such deviated patterns typically correspond to samples of interest and are assigned different labels in different domains, such as outliers, anomalies, exceptions, or malware. Detecting anomalies in fast, voluminous streams of data is a formidable chal- lenge. This paper presents a novel, generic, real-time distributed anomaly detection framework for heterogeneous streaming data where anomalies appear as a group. We have developed a distributed statistical approach to build a model and later use it to detect anomaly. As a case study, we investigate group anomaly de- tection for a VMware-based cloud data center, which maintains a large number of virtual machines (VMs). We have built our framework using Apache Spark to get higher throughput and lower data processing time on streaming data. We have developed a window-based statistical anomaly detection technique to detect anomalies that appear sporadically. We then relaxed this constraint with higher accuracy by implementing a cluster-based technique to detect sporadic and continuous anomalies. We conclude that our cluster-based technique out- performs other statistical techniques with higher accuracy and lower processing time.

Apache Spark↗

Detecting outliers in streaming time series data from ARM distributed sensors

The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliers through a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site and validated against existing database of known data quality issues.

Lu, Yuping↗

Final Report: Sampling-Based Algorithms for Estimating Structure in Big Data.

The purpose of this project was to develop sampling-based algorithms to discover hidden struc- ture in massive data sets. Inferring structure in large data sets is an increasingly common task in many critical national security applications. These data sets come from myriad sources, such as network traffic, sensor data, and data generated by large-scale simulations. They are often so large that traditional data mining techniques are time consuming or even infeasible. To address this problem, we focus on a class of algorithms that do not compute an exact answer, but instead use sampling to compute an approximate answer using fewer resources. The particular class of algorithms that we focus on are streaming algorithms , so called because they are designed to handle high-throughput streams of data. Streaming algorithms have only a small amount of working storage - much less than the size of the full data stream - so they must necessarily use sampling to approximate the correct answer. We present two results: * A streaming algorithm called HyperHeadTail , that estimates the degree distribution of a graph (i.e., the distribution of the number of connections for each node in a network). The degree distribution is a fundamental graph property, but prior work on estimating the degree distribution in a streaming setting was impractical for many real-world application. We improve upon prior work by developing an algorithm that can handle streams with repeated edges, and graph structures that evolve over time. * An algorithm for the task of maintaining a weighted subsample of items in a stream, when the items must be sampled according to their weight, and the weights are dynamically changing. To our knowledge, this is the first such algorithm designed for dynamically evolving weights. We expect it may be useful as a building block for other streaming algorithms on dynamic data sets.

97 MATHEMATICS AND COMPUTING↗

Code division multiple access signaling for modulated reflector technology

A method and apparatus for utilizing code division multiple access in modulated reflectance transmissions comprises the steps of generating a phase-modulated reflectance data bit stream; modifying the modulated reflectance data bit stream; providing the modified modulated reflectance data bit stream to a switch that connects an antenna to an infinite impedance in the event a "+1" is to be sent, or connects the antenna to ground in the event a "0" or a "-1" is to be sent.

97 MATHEMATICS AND COMPUTING↗