Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 235 records · Page 13
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

An Accurate and Automated Convective Vortex Detection Method for Long-Duration Infrasound Microbarometer Data

Heating of the surficial layer of the atmosphere often generates convective vortices, known as “dust devils” when they entrain visible debris. Convective vortices are common on both Earth and Mars, where they affect the climate via dust loading, contribute to wind erosion, impact the efficiency of photovoltaic systems, and potentially result in injury and property damage. However, long-duration terrestrial convective vortex activity records are rare. We have developed a high-precision and high-recall method to extract convective vortex signatures from infrasound microbarometer data streams. The techniques utilizes a wavelet-based detector to capture potential events and then a template matching system to extract the duration of the vortex. Since permanent and temporary infrasound sensors networks are present throughout the globe (many with open data), our method unlocks a vast new convective vortex dataset without requiring the deployment of specialized instrumentation.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

INITIAL STUDIES OF CAVITY FAULT PREDICTION AT JEFFERSON LABORATORY

The Continuous Electron Beam Accelerator Facility (CEBAF) at Jefferson Laboratory is a CW recirculating linac that utilizes over 400 superconducting radio-frequency (SRF) cavities to accelerate electrons up to 12 GeV through 5-passes. Recent work has shown that, given RF signals from a cavity during a fault as input, machine learning approaches can accurately classify the fault type. In this paper we report on initial results of predicting a fault onset using only data prior to the failure event. A data set was constructed using time-series data immediately before a fault (’unstable’) and 1.5 seconds prior to a fault (’stable’) gathered from over 5,000 saved fault events. The data was used to train a binary classifier. The results gave key insights into the behavior of several fault types and provided motivation to investigate whether data prior to a failure event could also predict the type of fault. We discuss our method using a sliding window approach and report on initial results. Recent modifications to the low-level RF control system will provide access to streaming signals and we outline a path forward for leveraging deep learning on streaming data

Vidyaratne, Lasitha↗
DOE OSTI2021

INITIAL STUDIES OF CAVITY FAULT PREDICTION AT JEFFERSON LABORATORY

The Continuous Electron Beam Accelerator Facility (CEBAF) at Jefferson Laboratory is a CW recirculating linac that utilizes over 400 superconducting radio-frequency (SRF) cavities to accelerate electrons up to 12 GeV through 5-passes. Recent work has shown that, given RF signals from a cavity during a fault as input, machine learning approaches can accurately classify the fault type. In this paper we report on initial results of predicting a fault onset using only data prior to the failure event. A data set was constructed using time-series data immediately before a fault (’unstable’) and 1.5 seconds prior to a fault (’stable’) gathered from over 5,000 saved fault events. The data was used to train a binary classifier. The results gave key insights into the behavior of several fault types and provided motivation to investigate whether data prior to a failure event could also predict the type of fault. We discuss our method using a sliding window approach and report on initial results. Recent modifications to the low-level RF control system will provide access to streaming signals and we outline a path forward for leveraging deep learning on streaming data

Vidyaratne, Lasitha↗
DOE OSTI2021

Data Analysis Approach for Large Data Volumes in a Connected Community

Recent advancements within smart neighborhoods where utilities are enabling automatic control of appliances such as heating, ventilation, and air conditioning (HVAC) and water heater (WH) systems are providing new opportunities to minimize energy costs through reduced peak load. This requires systematic collection, storage, management, and in-memory processing of large volumes of streaming data for fast performance. In this paper, we propose a multi-tier layered IoT software framework that enables effective descriptive and predictive data analysis for understanding live operation of the neighborhood, fault identification, and future opportunities for further optimization of load curves. We then demonstrate how we achieve live situational awareness of the connected neighborhood through a suite of visualization components. Finally, we discuss a few analytic dashboards that address questions such as peak load reductions obtained due to optimization, customer preference for automatic control of appliances (do they override the automatic control of HVAC?, etc.). 1 1 This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Chinthavali, Supriya↗
DOE OSTI2025

Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows

The evolving landscape of scientific computing requires seamless transitions from experimental to production HPC environments for interactive workflows. This paper presents a structured transition pathway developed at OLCF that bridges the gap between development testbeds and production systems. We address both technological and policy challenges, introducing frameworks for data streaming architectures, secure service interfaces, and adaptive resource scheduling for time-sensitive workloads and improved HPC interactivity. Our approach transforms traditional batch-oriented HPC into a more dynamic ecosystem capable of supporting modern scientific workflows that require near real-time data analysis, experimental steering, and cross-facility integration.

Etz, Brian [ORNL] (ORCID:0000000208554863)↗
DOE OSTI2024

Atmospheric Radiation Measurement (ARM) airborne field campaign data products between 2013 and 2018

Airborne measurements are pivotal for providing detailed, spatiotemporally resolved information about atmospheric parameters and aerosol and cloud properties, thereby enhancing our understanding of dynamic atmospheric processes. For 30 years, the US Department of Energy (DOE) Office of Science supported an instrumented Gulfstream 1 (G-1) aircraft for atmospheric field campaigns. Data from the final decade of G-1 operations were archived by the Atmospheric Radiation Measurement (ARM) Data Center and made publicly available at no cost to all registered users. To ensure a consistent data format and to improve the accessibility of the ARM airborne data, an integrated dataset was recently developed covering the final 6 years of G-1 operations (2013 to 2018, https://doi.org/10.5439/1999133; Mei and Gaustad, 2024). The integrated dataset includes data collected from 236 flights (766.4 h), which covered the Arctic, the US Southern Great Plains (SGP), the US West Coast, the eastern North Atlantic (ENA), the Amazon Basin in Brazil, and the Sierras de Córdoba range in Argentina. These comprehensive data streams provide much-needed insight into spatiotemporal variability in the thermodynamic quantities and aerosol and cloud properties for addressing essential science questions in Earth system process studies. This paper describes the DOE ARM merged G-1 datasets, including information on the acquisition, data collection challenges and future potentials, and quality control processes. It further illustrates the usage of this merged dataset to evaluate the Energy Exascale Earth System Model (E3SM) with the Earth System Model Aerosol–Cloud Diagnostics (ESMAC Diags) package.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Parallel Algorithms for Computing the Tensor-Train Decomposition

The tensor-train (TT) decomposition expresses a tensor in a data-sparse format used in molecular simulations, high-order correlation functions, and optimization. In this paper, we propose four parallelizable algorithms that compute the TT format from various tensor inputs: (1) Parallel-TTSVD for traditional format, (2) PSTT and its variants for streaming data, (3) Tucker2TT for Tucker format, and (4) TT-fADI for solutions of Sylvester tensor equations. We provide theoretical guarantees of accuracy, parallelization methods, scaling analysis, and numerical results. For example, for a d-dimension tensor in $\mathbb{R}$ $n\times∙∙∙$$\times$$n$ a two-sided sketching algorithm PSTT2 is shown to have a memory complexity of $O(n^{[d/2]})$, improving upon $O(n^{d—1})$ from previous algorithms.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Improving an Acoustic Vehicle Detector Using an Iterative Self-Supervision Procedure

In many non-canonical data science scenarios, obtaining, detecting, attributing, and annotating enough high-quality training data is the primary barrier to developing highly effective models. Moreover, in many problems that are not sufficiently defined or constrained, manually developing a training dataset can often overlook interesting phenomena that should be included. To this end, we have developed and demonstrated an iterative self-supervised learning procedure, whereby models are successfully trained and applied to new data to extract new training examples that are added to the corpus of training data. Successive generations of classifiers are then trained on this augmented corpus. Using low-frequency acoustic data collected by a network of infrasound sensors deployed around the High Flux Isotope Reactor and Radiochemical Engineering Development Center at Oak Ridge National Laboratory, we test the viability of our proposed approach to develop a powerful classifier with the goal of identifying vehicles from continuously streamed data and differentiating these from other sources of noise such as tools, people, airplanes, and wind. Using a small collection of exhaustively manually labeled data, we test several implementation details of the procedure and demonstrate its success regardless of the fidelity of the initial model used to seed the iterative procedure. Finally, we demonstrate the method’s ability to update a model to accommodate changes in the data-generating distribution encountered during long-term persistent data collection.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2020

Advanced Laboratory and Field Arrays: Evaluating Sampling Techniques for MHK Biological Monitoring (Task 6)

The overall goal of the task was to identify cost effective biological sampling techniques for MHK environmental monitoring. Protected, demersal, and pelagic fish and selected nektonic invertebrates were surveyed using capture and remote sensing techniques at the PacWave sites. The performance of capture and remote sensing monitoring techniques were evaluated and generic nekton monitoring indices were developed for MHK technologies and sites. In parallel to data collections at the PacWave site, the ability of regression models to characterize, detect, and predict change in acoustic data was evaluated using acoustic data collected in Admiralty Inlet, WA, during a biological monitoring study of the proposed SnoPud tidal turbine project site. Expected outcomes of these efforts included an evaluation of instrumentation and techniques used to monitor biological variability; identification of data streams that can be used to detect and quantify change; and sampling requirements to ensure detection of change in monitored variables.

13 HYDRO ENERGY↗
DOE OSTI2025

COMPASS-FME Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) Experiment Level 1 Sensor Data v2-1

This is the version 2-1 Level 1 (L1) data release for COMPASS-FME environmental sensors located at our Terrestrial Ecosystem Manipulation to Probe the Effects of Storm Treatments (TEMPEST) experimental site. This manipulative, ecosystem-scale TEMPEST experiment addresses the potential for freshwater and estuarine-water disturbance events to alter tree function, species composition, and ecosystem processes in a deciduous coastal forest in MD, USA. The experiment uses a large-unit (2000 m2), un-replicated experimental design, with three 50 m × 40 m plots serving as control, freshwater, and estuarine-water treatments. L1 data are close to raw, but are units-transformed and have out-of-instrument-bounds, out-of-service, and outlier flags added. Duplicates and missing data are removed but otherwise these data are not filtered, and have not been subject to any additional algorithmic or human QA/QC. Any scientific analyses of L1 data should be performed with care. **This dataset will be updated quarterly with new data for the duration of the project** This dataset includes: - An overall dataset README file that describes the current version, gives citation and contact information, etc. - Site- and year-specific folders, each holding variable-specific CSV (comma separated value) data files for each site and plot in that year. - Metadata files within each site-year folder provide full information on data units, expected ranges, contact information, detailed flood times, as well as a general description of the site. - Environmental sensor types that appear in the data files include weather (ClimaVUE50, CS, RM Young, and LI instruments in the graphs below); soil conditions (TEROS12); soil redox state (Redox); groundwater variables (AquaTROLL200 and AquaTROLL600); open water sondes (Exo); tree sap velocity (Sapflow); and system voltage and state (Datalogger). Data are normally logged every 15 minutes. Please see v2-1 TEMPEST L1 Sensor Package Quick Start.pdf for detailed information on data package structure, temporal coverage, and versioning. The TEMPEST flood events occurred on the following dates. They lasted for ~10 hours each day and delivered ~80,000 gallons to each plot; many data streams are available at 1 or 5 minute frequency during these periods. * Tests: Aug 25 (fresh plot) and Sep 9 (salt plot), 2021 * TEMPEST 1: June 22, 2022 * TEMPEST 2: June 6-7, 2023 * TEMPEST 3: June 11-13, 2024 This dataset was updated 2026-03-12: (i) data now go through 2025-12-31 (previous end was 2025-06-30) and (ii) dataset and file names updated to “…v2-1” (previously was “v2-0”).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Towards a Stronger Foundation: Digitizing Commercial Buildings with Brick to Enable Portable Advanced Applications

Most large commercial buildings have digital controls for their heating, ventilation, and air-conditioning (HVAC) and lighting systems with the potential to implement advanced control strategies and data analytics. However, advanced control strategies and data analytics are rarely deployed at scale due to non-standard naming conventions and heterogenous building configurations. Semantic metadata standards, like Brick, show promise to proliferate these applications across many buildings, but they have not been widely adopted by industry due to barriers such as perceived risk and unfamiliarity with the technology. This paper describes the workflow we established and evaluated while using it to develop over ten Brick models of existing buildings. Through this process, we observed that digitizing existing commercial buildings is a cost and labor-intensive effort in which understanding the buildings’ data streams is the major bottleneck. Yet, we conclude this investment is worthwhile since various use case applications such as fault detection and diagnostics, thermal comfort analysis, and HVAC control optimization can utilize the same Brick model. The paper also explores the challenges and lessons learned we encountered while creating these data models, such as: 1) difficulties in finding metadata descriptions and relationships for existing buildings; 2) handling missing concepts in the schema needed to model a building; 3) lack of guidance on how to structure the data model or how much detail to include; 4) unfamiliarity with technologies, which makes the learning curve steep for applications developers. Finally, we also describe future directions for semantic metadata research and development to make such transformative technologies more accessible to practitioners.

Roa, Carlos Duarte↗
DOE OSTI2021

Data-Driven Adaptive Damping Controller for Wind Power Plants with Doubly-Fed Induction Generators

This paper presents an adaptive damping controller for wind power plants in which the turbines are equipped with doubly-fed induction generators. The controller is designed to respond to an input control signal that is triggered according to the system operating conditions. A processing unit continuously estimates the electromechanical modes of oscillation based on real-time streaming data acquired from a phasor measurement unit that is strategically positioned on the grid. The decision to trigger (or not trigger) the control signal is automatic, based on the relative damping of the dominant mode. The modes are estimated using the dynamic mode decomposition algorithm with time-delay embedding. Numerical simulations performed on the two-area system demonstrate that the proposed controller enhances the rotor angle stability for both small-signal and large disturbances, and is adaptive to changing grid conditions.

damping controller↗
DOE OSTI2021

Data-Driven Adaptive Damping Controller for Wind Power Plants with Doubly-Fed Induction Generators: Preprint

This paper presents an adaptive damping controller for wind power plants in which the turbines are equipped with doubly-fed induction generators. The controller is designed to respond to an input control signal that is triggered according to the system operating conditions. A processing unit continuously estimates the electromechanical modes of oscillation based on real-time streaming data acquired from a phasor measurement unit that is strategically positioned on the grid. The decision to trigger (or not trigger) the control signal is automatic, based on the relative damping of the dominant mode. The modes are estimated using the dynamic mode decomposition algorithm with time-delay embedding. Numerical simulations performed on the two-area system demonstrate that the proposed controller enhances the rotor angle stability for both small-signal and large disturbances, and is adaptive to changing grid conditions.

61 RADIATION PROTECTION AND DOSIMETRY↗
DOE OSTI2025

A co-registered in-situ and ex-situ dataset from wire arc additive manufacturing process

Recent progress in sensing techniques and data analytics tools have significantly accelerated the development of Wire Arc Additive Manufacturing (WAAM) systems. This data-centric approach emphasizes leveraging sensor data available throughout the production process to optimize performance. Integration of extensive data analysis provides opportunities for improving precision, reducing waste, and enhancing the quality of produced parts. This method relies on AI/ML models and optimization techniques, which are developed using the data collected from various sources, including in-situ sensors, ex-situ imaging, and manufacturing process parameters. The quality and diversity of this data, along with the alignment between different data streams (achieved through spatiotemporal registration) are critical for the successful development of AI/ML and optimization models. In this work, we present a spatiotemporally registered dataset generated during the WAAM process of deposition of a rectangular block. The dataset includes a comprehensive description of the deposition process, process parameters, welding characteristics and acoustic data collected in-situ, and X-Ray Computed Tomography data of the build.

42 ENGINEERING↗
DOE OSTI2025

Classifying thermodynamic cloud phase using machine learning models

Vertically resolved thermodynamic cloud-phase classifications are essential for studies of atmospheric cloud and precipitation processes. The Department of Energy (DOE) Atmospheric Radiation Measurement (ARM) Thermodynamic Cloud Phase (THERMOCLDPHASE) value-added product (VAP) uses a multi-sensor approach to classify the thermodynamic cloud phase by combining lidar backscatter and depolarization, radar reflectivity, Doppler velocity, spectral width, microwave-radiometer-derived liquid water path, and radiosonde temperature measurements. The measured pixels are classified as ice, snow, mixed phase, liquid (cloud water), drizzle, rain, and liq_driz (liquid+drizzle). We use this product as the ground truth to train three machine learning (ML) models to predict the thermodynamic cloud phase from multi-sensor remote sensing measurements taken at the ARM North Slope of Alaska (NSA) observatory: a random forest (RF), a multi-layer perceptron (MLP), and a convolutional neural network (CNN) with a U-Net architecture. Evaluations against the outputs of the THERMOCLDPHASE VAP with 1 year of data show that the CNN outperforms the other two models, achieving the highest test accuracy, F1 score, and mean intersection over union (IOU). Analysis of ML confidence scores shows that ice, rain, and snow have higher confidence scores, followed by liquid, while mixed, drizzle, and liq_driz have lower scores. Feature importance analysis reveals that the mean Doppler velocity and vertically resolved temperature are the most influential data streams for ML thermodynamic cloud-phase predictions. Lidar measurements exhibit lower feature importance due to rapid signal attenuation caused by the frequent presence of persistent low-level clouds at the NSA site. The ML models' generalization capacity is further evaluated by applying them at another Arctic ARM site in Norway using data taken during the ARM Cold-Air Outbreaks in the Marine Boundary Layer Experiment (COMBLE) field campaign. The models demonstrated similar performance to that observed at the NSA site. Finally, we evaluate the ML models' response to simulated instrument outages and signal degradation and show that a CNN U-Net model trained with input channel dropouts performs better when input fields are missing.

ARM Aerial Facility↗
DOE OSTI2022

100 Gb/s High Throughput Serial Protocol (HTSP) for data acquisition systems with interleaved streaming

Demands on Field-Programmable Gate Array (FPGA) data transport have been increasing over the years as frame sizes and refresh rates increase. As the bandwidths requirements increase the ability to implement data transport protocol layers using "soft" programmable logic becomes harder and start to require harden IP blocks implementation. To reduce the number of physical links and interconnects, it is common for data acquisition systems to require interleaving of streams on the same link (e.g. streaming data and streaming register access). Further, this paper presents a way to leverage existing FPGA harden IP blocks to achieve a robust, low latency 100 Gb/s point-to-point link with minimal programmable logic overhead geared towards the needs of data acquisition systems with interleaved streaming requirements.

47 OTHER INSTRUMENTATION↗
DOE OSTI2023

HopPyBar

HopPyBar is a python program to import, analyze, and export split-Hopkinson pressure bar (SHPB, also known as Kolsky bar) data. Traditional analysis offers a black box approach, where input data is converted to analyzed output by performing a series of calculations without user involvement. This program serves as a developmental platform to "white box" the data analysis process. Data streams can be captured (in-situ) to enable advanced or unconventional analyses, statistics, and comparisons. Additionally, the program is geared towards the standardized forms of input and output used at LANL to streamline analysis, but the open nature of the program makes additional input/output schemes straightforward to add. General workflow will import SHPB data in one of a number of formats, identify relevant portions of data signals, and convert to stress-strain-strain rate to show material behavior as a function of dynamic testing.

Morrow, Benjamin↗