Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

Developing a Decision Support Engine to Enable Irrigation Modernization - Poster

Irrigation systems in the United States are some of the oldest continually utilized infrastructure in existence today, with some systems exceeding 100 years in age. They are operated to meet farming demands but are managed through a balance of varying influence: policies limiting water usage, stakeholder interests, and environmental impacts. Irrigation modernization is defined as a set of activities that update and improve existing irrigation systems, including, but not limited to improving water quantity, development of distributed energy resources for surrounding communities, ecosystem services, and improved agricultural yields. Modernizing an existing irrigation system can enable stakeholders to combat changing environmental and population demands but is difficult because the complexity involved in determining the potential benefits and consequences of irrigation modernization is high. We are combining a large amount of various geospatial, tabular, and temporal data with subject matter expertise into a decision support engine that will enable stakeholders to determine the benefits and consequences of irrigation modernization in their irrigation systems. A web-based GIS will allow the user to construct the modifications out of a palette of modernization options, which will be sent to the analytics engine for computations, and back to the web client for a graphical display and comparison of relevant metrics. Our development process involves four phases: 1) identify mechanisms of modernization, 2) identify data requirements, data streams, first principles and applicable algorithms necessary to quantify modernization mechanisms, 3) create ‘modules’ for each modernization mechanism, these modules will form the decision support engine, each capable of performing independently but can also inform other modules when needed, 4) Merge the decision support engine with a user interface, capable of ingesting user inputs and returning insights into the impacts of a modernization project as they relate to economic, environmental, monetary, and energy generation. Once complete, it is our intention that this tool will be fundamental in irrigation modernization projects, providing a strong analytical basis from which stakeholders can quickly make informed decisions regarding project development.

13 HYDRO ENERGY↗
DOE OSTI2023

FPGA Architectures for Distributed ML Systems for Real-time Beam Loss De-blending

The Accelerator Real-time Edge AI for Distributed Systems (READS) project’s goal is to create a Artificial Intelligence (AI) system for real-time beam loss de-blending within the accelerator enclosure, which houses two accelerators: the Main Injector (MI) and the Recycler Ring (RR). In periods of joint operation, when both machines contain high intensity beam, radioactive beam losses from MI and RR overlap on the enclosure’s beam loss monitoring Beam Loss Monitor (BLM) system, making it difficult to attribute those losses to a single machine. Incorrect diagnoses result in unnecessary downtime that incurs both financialand experimental cost. The ML system will automatically disentangle each machine’s contributions to those measured losses, while not disrupting the existing operations-critical functions of the BLM system. This paper will focus on the evolution of the architectures, which provided the high-frequency, low-latency collection of synchronized data streams to make real-time inferences. The ML models, used for learning both local and global machine signatures and producing high quality inferences based on raw BLM loss measurements, will only be discussed at a high-level.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

Single Carrier Modulation Resource Allocation for Massive MIMO with Virtual Antennas

Today’s Massive MIMO cellular operation is dominated by orthogonal frequency division multiplexing (OFDM) modulation. One of the advantages of OFDM is the flexibility to carve up the available spectrum into resource blocks (RBs) that can operate adjacent to one another. Massive MIMO adds spatial multiplexing layers on top of the RBs, enabling the simultaneous operation of dozens of UEs. Cyclic prefixed single carrier modulation (CP-SCM) is an alternative to OFDM that also benefits from massive MIMO and is useful as an OFDM alternative in specific scenarios. However, until recently, no resource allocation method has been proposed for CP-SCM. In this paper, we present a CP-SCM resource allocation scheme using virtual antennas in a massive MIMO time domain duplexed scenario. By creating several virtual antennas for each physical antenna, the benefits of massive MIMO are achieved with a smaller physical antenna count. Resources are quantized into data streams, and each user can be assigned a variable number of simultaneous streams. This paper presents a detailed development and analysis of multi-stream processing (MSP) for both uplink detection and downlink precoding. We also introduce heterogeneous MSP, where CP-SCM and OFDM signals can be processed in the same MSP framework.

5G and beyond communications↗
DOE OSTI2024

A Multi-Dimensional Benefit Assessment of Automated Mobility Platforms (AMP) for Large Facilities

Automated Mobility Platform (AMP) is a general term for intelligently managed passenger mobility systems that leverage automation and electrification to improve system performance and sustainability through the integration of real-time data streams with advanced decision-making processes tailored for use in large facilities. Using airports as an example, the rapid growth in air travel demand has led to facility expansions and congested terminals, which directly impacts equity (e.g., increased challenges for passengers with reduced mobility [PRMs]) and sustainability - both of which are important metrics often neglected during the engineering design process. Therefore, to evaluate systems and inform critical decisions more effectively, a holistic evaluation framework is proposed that includes equity, sustainability, and infrastructure perspectives - referred to as a multidimensional benefit assessment. More specifically, the following analysis focuses on: (1) mobility, with emphasis on travel time and accessibility within an airport, contrasting existing systems with the proposed AMP system; (2) equity analysis, specifically to the PRM community, but with an eye to the whole of society; (3) energy consumption and greenhouse gas (GHG) emissions associated with intra-airport mobility, comparing and contrasting modes as well as impacts on overall airport operations efficiency; and (4) fundamental changes in building design enabled by AMPs for larger and more flexible, functional, and energy-efficient structures.

ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2022

Improved Line Outage Detection in Transmission Systems with Few PMUs

Unlike transmission systems, distribution systems historically lack enough measurements, making their real-time monitoring almost impossible. Recent deployment of diverse types of devices such as phasor measurement units (PMUs), smart meters, solar inverters and weather information sensors opens up new ways of monitoring these systems, with the assistance of customized machine learning (ML) applications. The paper describes a grid-model-informed machine learning (ML) tool which integrates heterogeneous data streams and creates synchronous measurement snapshots to be used by a hybrid robust state estimator (SE) which provides not only accurate state estimates but also real-time feedback for ML model refinement. Improved monitoring performance due to the use of developed computational framework is experimentally observed by simulated scenarios on an electric utility’s distribution system.

Distribution systems, graph learning, machine lear↗
DOE OSTI2025

The Sensor Dilemma in Intelligent Transportation Systems: Evaluating Radar, Lidar and Camera: Preprint

Intelligent transportation systems (ITS) are at the forefront in advancing the way we interact with and perceive the transportation network. This revolution is fueled by the significant advancement in sensor perception technologies such as radar, lidar, and video imaging, which are the most popular modalities for ITS. Real-time perception data from these sensors allow intelligent infrastructure-side decision-making to improve the energy, efficiency, and safety at traffic intersections. As traffic departments across the United States transition from traditional loop detectors and emulators and embrace newer technologies, they are often left with a dilemma in choosing a sensor technology for infrastructure-based perception that is reliable, inexpensive, and easy to set up and that has robust performance in varying weather conditions. However, choosing a sensor that checks all these boxes is not straightforward, as every sensor type has unique benefits and drawbacks. Radar is excellent at detecting long-range vehicles and weather resistance but lacks high resolution. Lidar is expensive and weather-sensitive, while cameras provide rich visual data at a low cost but are constrained by lighting and visibility. This study examines radar, lidar, and camera sensor capabilities to ascertain whether any of these qualifies as the "best" sensor for ITS perception. While no single sensor can meet all the demands of ITS, a hybrid approach combining multiple sensor modalities like radar, lidar, and cameras offers the most robust solution for enhancing the safety and efficiency of ITS. Through this evaluation, we hope to draw attention to the necessity of the National Renewable Energy Laboratory's infrastructure perception and control framework, which presents a multisensor track data fusion engine to assimilate multiple data streams in order to provide robust and reliable perception.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2026

Neutrino beam bunch structure reconstruction with precision timing in the ICARUS liquid argon time projection chamber

The ICARUS detector has been operating smoothly since 2021 as the far detector in the Short Baseline Neutrino (SBN) program at Fermilab, collecting neutrino interactions from both the Booster Neutrino Beam (BNB) and off-axis from the Neutrinos at the Main Injector (NuMI) beam. Analysis of neutrino interactions in ICARUS requires mitigation of substantial cosmogenic backgrounds. This is achieved by using an external Cosmic Ray Tagger (CRT) and a Photomultiplier Tube (PMT) system embedded in the liquid argon. The intrinsic neutrino beam bunch structure, inherited time structure from the Radio Frequency (RF) system used to accelerate the protons, can be resolved at the ICARUS detector using precise timing information. Located at shallow depth, ICARUS is exposed to a high flux of cosmic rays that can be mistaken for neutrino interactions. To mitigate this background, the CRT and a 3-meter-thick concrete overburden were installed. To better model backgrounds, ICARUS makes use of an overlay technique where simulated neutrino events are superimposed on detector beam-off data. PMTs installed within a Time Projection Chamber detect argon scintillation light emitted by high energy charged particles passing through the chamber and provide the event timing of neutrino interactions. The nanosecond-level beam bunch structure is reconstructed with the PMT system and can be used to further understand backgrounds and enhance neutrino physics capabilities. In this thesis, I will discuss background mitigation techniques using precision timing and present a novel technique to select neutrino events from our unbiased data stream using the beam bunch structure.

Heggestuen, Anna [Colorado State U.] (ORCID:000000↗
DOE OSTI2026

EXCLUSIVE NEUTRAL PION ELECTROPRODUCTION CROSS SECTION MEASUREMENTSWITHANEUTRALPARTICLE SPECTROMETER

Deep Virtual Compton Scattering (DVCS), the exclusive electron-proton scattering process ep ¿e'p'¿, provides access to generalized parton distributions (GPDs), which correlate information about the longitudinal momentum and transverse spatial structure of quarks inside the nucleon. Experiment E12-13-010 in Hall C at Jefferson Lab was designed to take high-precision measurements of the DVCS cross section over an extended kinematic range using the newly commissioned Neutral Particle Spectrometer (NPS). The NPS features a high-resolution electromagnetic calorimeter and a streaming data acquisition system optimized for operation at high luminosities. This thesis presents the detector and analysis work carried out to support the NPS DVCS program. In particular, it focuses on the hardware design, calibration, and performance of the calorimeter. A development of a waveform reconstruction analysis of the calorimeter signals enabled improved extraction of pulse amplitudes and times. The waveform analysis was also extended to operate in a multithreaded environment, substantially reducing processing time for large datasets. Analysis of exclusive neutral pion electroproduction events in the calorimeter gives a strong validation of the calorimeter’s performance and resolution. Together these developments establish a foundation for future analyses and extraction of the DVCS cross section and its use in constraining the GPDs.

Kerver, Mitchell [Old Dominion Univ., Norfolk, VA ↗
DOE OSTI2022

Data from: “Significant stream chemistry response to temperature variations in a high-elevation mountain watershed”

High-elevation mountain regions, central to global freshwater supply, are experiencing more rapid warming than low-elevation locations. High-elevation streams are therefore potentially critical indicators for earth system and water chemistry response to warming. Here we present concerted hydroclimatic and biogeochemical data from Coal Creek, Colorado in the central Rocky Mountains at elevations of 2700 to 3700 m, where air temperatures have increased by about 2 °C since 1980. We analyzed water chemistry every other day from 2016 to 2019. Water chemistry data indicate distinct responses of different solutes to inter-annual hydroclimatic variations. Specifically, the concentrations of solutes from rock weathering are stable inter-annually. Solutes that are active in soils, including dissolved organic carbon, vary dramatically, with double to triple peak concentrations occurring during snowmelt and in warm years. We advocate for consistent and persistent monitoring of high elevation streams to record early glimpse of earth surface response to warming.1) Coal Creek stream were sampled and measured to answer the question if there is a significant stream chemistry response to climate change in high-elevation watershed2) The file contains data regarding discharge, water temperature and stream chemistry data for 2016-2019 year. Stream chemistry includes dissolved organic and inorganic carbon, chloride, nutrients, cations. The data collection efforts were carried out by the DOE-funded "Advancing a Watershed Hydro-Biogeochemical Theory: Linking Water Travel Time and Reaction Rates Under Changing Climate" project and Watershed Function SFA.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Investigating the impacts of solid phase extraction on dissolved organic matter optical signatures and the pairing with high‐resolution mass spectrometry data across a freshwater stream network

Abstract Advancing our understanding of dissolved organic matter (DOM) chemistry in aquatic systems necessitates the integration of data streams from multiple analytical platforms. Some measurements require pretreatment with solid phase extraction (SPE), while others are performed directly on whole water samples. Evidence has suggested that SPE will be biased against select DOM fractions, leading to concerns over the ability to establish data linkages across platforms with variable needs for SPE pretreatment, such as those from optical measurements and those that provide high‐resolution molecular information. Here, we directly addressed this concern by assessing the impact of SPE on DOM optical properties through excitation–emission matrices with parallel factor analysis (PARAFAC) for 47 samples across a stream network within a single watershed reflective of variable DOM sources. PARAFAC data was further paired with molecular information obtained by Fourier transform ion cyclotron resonance mass spectrometry (FTICR‐MS). A comparison of PARAFAC models first revealed no systematic qualitative differences in major components between whole water DOM and DOM isolated by SPE (SPE‐DOM); however, quantitative biases against select components were observed. Further linkages with FTICR‐MS data revealed that the molecular fingerprint associated with each PARAFAC component was consistent between the whole water DOM and SPE‐DOM. Our results suggest that bulk scale linkages across these analytical platforms could be inferred irrespective of the observed quantitative biases resulting from SPE for samples within this example watershed. This work represents a key step toward the systematic evaluation of linkages between optical and high‐resolution mass spectrometry datasets in freshwater lotic environments.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2022

Large Scale Caching and Streaming of Training Data for Online Deep Learning

The training of deep neural network models on large data remains a difficult problem, despite progress towards scalable techniques. In particular, there is a mismatch between the random but predetermined order in which AI flows select training samples and the streaming I/O patterns for which traditional HPC data storage (e.g., parallel file systems) are designed. In addition, as more data are obtained, it is feasible neither simply to train learning models incrementally, due to catastrophic forgetting (i.e., bias towards new samples), nor to train frequently from scratch, due to prohibitive time and/or resource constraints. In this paper, we study data management techniques that combine caching and streaming with rehearsal support in order to enable efficient access to training samples in both offline training and continual learning. We revisit state-of-art streaming approaches based on data pipelines that transparently handle prefetching, caching, shuffling, and data augmentation, and discuss the challenges and opportunities that arise when combining these methods with data-parallel training techniques. We also report on preliminary experiments that evaluate the I/O overheads involved in accessing the training samples from a parallel file system (PFS) under several concurrency scenarios, highlighting the impact of the PFS on the design of the data pipelines.

data pipelines↗
DOE OSTI2024

Kernelized approaches to streaming compression of scientific data

In this paper three algorithms are developed for the streaming compression of scientific data. The algorithms presented are reliant on the theory of vector-valued reproducing kernel Hilbert spaces and operator valued kernel. Further, the scientific data is modeled as a snapshot of time dependent vector field F(x, t) over a manifold M and the recovery of the data is framed as a learning problem. These processes are then appropriately modified and ana lyzed for the streaming scenario in which data is generated without the ability to revisit past entries.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Data and scripts associated with "Coupled primary production and respiration in a large river contrasts with smaller rivers and streams."

This data package is associated with the publication "Coupled primary production and respiration in a large river contrasts with smaller rivers and streams." in review at Limnology and Oceanography (Roley et al. 2023). This study focuses on understanding ecosystem metabolism for the Hanford Reach of the Columbia River in Washington state, a free-flowing stretch with a substantial discharge. Large rivers have been overlooked compared to small and medium rivers due to the challenges associated with measurements. Our study presents novel ways to address these challenges and highlights that metabolism patterns in large rivers differ from those observed in small-medium rivers and requires the application of knowledge and tools beyond those implemented for smaller rivers.This data package includes the data and R scripts for the analyses described in Roley et al. 2023. It includes dissolved oxygen and temperature data from a dissolved oxygen HOBO sensor, light data collected from the National Solar Radiation Database (https://nsrdb.nrel.gov/) and hydrologic variables estimated from the MASS-1 model (Niehus et al.; 2014). It also includes metabolism estimates (gross primary production, ecosystem respiration, and net ecosystem production) estimated via streamMetabolizer (Appling et al.; 2018). All analyses in the paper can be replicated with these data and scripts.The data package is comprised of one main data folder. The folder includes (1) file-level metadata (flmd); (2) a data dictionary (dd) for each data file; (3) data files; and (4) R scripts for metabolism estimates and data analysis. All files are .R, .csv, or .pdf.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2026

Signatures of a Tidally Induced Spiral Arm at the Anticenter of the Milky Way and a Kinematically Extended Anticenter Stream Using DESI Data Release 2

Using the Dark Energy Spectroscopic Instrument (DESI) Milky Way Survey, we examine the six-dimensional space of the anticenter region of the Milky Way stellar disk (150° < Galactic longitude < 220°) using 61,883 main-sequence turnoff stars. We focus on two well-known stellar overdensities in the anticenter: the Monoceros Ring (MRi) and Anticenter Stream (ACS). We find that the MRi overdensity has kinematic signatures consistent with a tidally induced spiral arm, a type of dynamic spiral arm created by an interaction with a satellite galaxy, most likely the Sagittarius dwarf spheroidal galaxy (Sgr). We use the kinematics of the MRi to calculate the two most recent passage times of Sgr, finding 0.25 ± 0.09 Gyr and 1.10 ± 0.23 Gyr from the present day. We validate that the ACS is kinematically decoupled from the MRi because they are moving in opposite radial and vertical directions. We find that the kinematics associated with the ACS extends beyond our defined overdensity. The features we see in the ACS region are likely part of a broader distribution of stars with the same kinematic signature as detected in other places, like the vertical wave in the outer disk and phase spiral.

Lambert, Mika [University of California, Santa Cru↗
DOE OSTI2024

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s -1 (87,000 frames s -1 ⁠), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

4D-STEM↗
DOE OSTI2024

Machine Learning for Joint Quality Control

The use of lightweight material combinations has been highly demanded in manufacturing automotive structures. However, making robust dissimilar material joints of such lightweight materials is still challenging. A significant barrier to achieving high-quality and repeatable joint performance is a deficient understanding of the relationship between the welding process, joint attributes, and joint performance. In this context, welding factors refer to material, equipment, environment, and process parameters, while joint features comprise specific microstructural attributes of the weld such as nugget size, heat affected zone (HAZ) topology, intermetallic layer thickness, and sheet thickness reduction. Joint performance is quantified in terms of strength (e.g., tensile shear, coach peel, cross-tension), weld size, and hardness, among other factors. While there have been many attempts to establish this process-structure-property relationship by developing a model derived from the associated physics and first principles, the complexity of the joining processes compounded by the complex interactions with different materials in an automotive assembly line environment, has hindered the usefulness of such attempts. The complexity is further exacerbated using different stacking materials, especially comprising dissimilar material combinations. In practice, the common approach has been the laborious process of creating welds, characterizing them, and then physically testing them through experimentation. With the emergence of artificial intelligence (AI) methods, an alternative pathway to eliciting the desired process-structure-property relationship at an accelerated pace is to use a data-driven approach by employing machine-learning (ML) techniques. This approach is benefitted by the availability of large streams of data, generated through years of research and testing by original equipment manufacturers, in the form of material, process, environmental, equipment, microstructural, and bulk-scale performance information from multimodal, multiscale sensors making measurements from laboratory-scale to production-scale processes. During Phase I efforts, which ended in fiscal year (FY) 2021, the Oak Ridge National Laboratory and Pacific Northwest National Laboratory (ORNL/PNNL) team demonstrated the effectiveness of different ML/AI frameworks in modeling complex relationships between resistance spot welding (RSW) process parameters, weld attributes, and joint properties using a subset of data from General Motors (GM). In FY 2022, the project team further refined and expanded their respective ML models to analyze additional welds with new weld stack-ups and materials to enhance the ML model predictive capability. ORNL extended its unified deep neural networks (DNN) ML training and prediction framework with new data streams of process parameters, and PNNL extended its model describing RSW process parameters’ associations with weld attributes. In FY 2023, the project team completed the development of the AI/ML architecture for analyzing aluminum/steel joints manufactured by GM via RSW and transitioned into the inline welding quality monitoring task for steel/steel RSW joints provided by GM.

36 MATERIALS SCIENCE↗