Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Automation for Grid Interconnected Laboratory Emulation

As computational capabilities improve, digital twins are becoming vital for evaluating equipment realistically in laboratories. This paper outlines a digital twin architecture for the power grid, employing electromagnetic transient (EMT) simulation alongside real-time simulation of power hardware and hierarchical control systems. EMT simulation occurs on a high-performance computing server for scalability. Additionally, the paper describes a workflow and real-time data streaming software facilitating connectivity among EMT simulation, hierarchical control systems, and power hardware. This software enables automated equipment connectivity in the laboratory for realistic evaluations, aiding in identifying necessary upgrades for both equipment control systems and the power grid.

Marthi, Phani Ratna Vanamali [ORNL] (ORCID:0000000↗
DOE OSTI2021

PMU-data-driven Event Classification in Power Transmission Grids

This paper presents an event classification in transmission grids. The convolutional neural network (CNN)-based classifier is proposed to capture the temporal similarity of time-synchronized data stream from phasor measurement units (PMUs). The proposed CNN is trained using Bayesian optimization to search for the best hyperparameters. The effectiveness of the proposed event classification is validated through the real-world dataset from the U.S. transmission grids. This dataset includes line outage, transformer outage, frequency, and oscillation events. The validation process also includes different PMU outputs, such as voltage magnitude, phase angle, current magnitude, frequency, and rate of change of frequency (ROCOF). The results show that ROCOF gives the best classification performance compared to other PMU outputs. In addition, it is shown that the classifier trained with a larger dataset has higher accuracy. Moreover, the superiority of the proposed method is validated through comparison with other state-of-the-art classification methods.

Niazazari, Iman↗
DOE OSTI2025

Free-Space Optical Channel Turbulence Prediction: A Machine Learning Approach

Channel turbulence is a formidable obstacle for freespace optical (FSO) communication. Anticipation of turbulence levels is highly important for mitigating disruptions but has not been demonstrated without dedicated, auxiliary hardware. We show that machine learning (ML) can be applied to raw FSO data streams to rapidly predict channel turbulence levels with no additional sensing hardware. FSO was conducted through a controlled channel in the lab under six distinct turbulence levels, and the efficacy of using ML to classify turbulence levels was examined. ML-based turbulence level classification was found to be > 98% accurate with multiple ML training parameters. Classification effectiveness was found to depend on the timescale of changes between turbulence levels but converges when turbulence stabilizes over about a one minute timescale.

Zobaer Islam, Md [School of Electrical and Compute↗
DOE OSTI2020

Simulating the Autonomous Future: A Look at Virtual Vehicle Environments and How to Validate Simulation Using Public Data Sets

The rapid evolution of autonomous vehicles (AVs) has exposed the need for fast-paced development and testing processes of a variety of perception, planning, and control algorithms. To expedite development, the AV industry and researchers leverage virtual vehicle environments to simulate a range of test scenarios that may otherwise be costly or difficult to conduct on a real test track. However, the various virtual environments may have different results depending on the fidelity of various simulation features, such as vehicle dynamics, sensor simulation, and environment recreation. Herein, this tutorial article examines a proposed framework for constructing, parameterizing, and validating a virtual vehicle environment using an existing AV data set. First, an overview of several open source and commercially available simulation tools, including their associated workflows, for scene and scenario creation is presented. Next, various open AV data sets are examined to inform the data set selection for the validation framework. Then, an example workflow of recreating a real-world scene from the selected data set in a simulation tool with various emulated sensors parameterized to match the data set is demonstrated. Finally, an example AV-perception algorithm is subjected to data streams from virtual and real-world environments and suggested metrics for analyzing the results are discussed.

42 ENGINEERING↗
DOE OSTI2021

Event Cause Analysis in Distribution Networks using Synchro Waveform Measurements

This paper presents a machine learning method for event cause analysis to enhance situational awareness in distribution networks. The data streams are captured using time-synchronized high sampling rates synchro waveform measurement units (SWMU). The proposed method is formulated based on a machine learning method, the convolutional neural network (CNN). This method is capable of capturing the spatiotemporal feature of the measurements effectively and perform the event cause analysis. Several events are considered in this paper to encompass a range of possible events in real distribution networks, including capacitor bank switching, transformer energization, fault, and high impedance fault (HIF). The dataset for our study is generated using the real time digital simulator (RTDS) to simulate real-world events. The event cause analysis is performed using only one cycle of the voltage waveforms after the event is detected. The simulation results show the effectiveness of the proposed machine learning-based method compared to the state-of-the-art classifiers.

Niazazari, Iman↗
DOE OSTI2022

Real-time Event Detection Using Rank Signatures of Real-world PMU Data

Timely detection of power system events is a crucial task, which can facilitate the implementation of remedial actions to improve reliability, resiliency, and security of the system. Meanwhile, the widespread deployment of phasor measurement units (PMUs) makes it possible to develop data-driven event detection techniques. However, relying purely on data without incorporating domain knowledge for the event detection task in power systems poses substantial security and stability risks due to issues associated with data misinterpretation and model accuracy. In this regard, we propose a real-time event detection method using real-world PMU data by incorporating domain knowledge to adequately capture the event signatures. Specifically, we track the change in rank signatures of PMU data to accurately localize the events. To optimize the detection process, we incorporate an offline Bayesian optimization algorithm to tune the parameters by efficiently searching for the best values. The experiments using the real-world PMU dataset from a U.S. interconnection show that the proposed event detection approach can efficiently detect the events from PMU data streams with high accuracy.

Ghasemkhani, Amir↗
DOE OSTI2024

Enabling DER visibility using a distributed dissemination network

The electrical grid is currently undergoing a series of rapid transformational changes that have resulted in the introduction of new actors and operational schemes that have fragmented the data and control planes. To help address the issue, this paper describes the implementation of a sensor-oriented, distributed data dissemination network that seeks to eliminate data silos. The implementation is based on the DGSS architecture previously described in [1]. The developed product seeks to facilitate the seamless integration of multi-operator, multi-origin, multi-domain sensor data by using a distributed systems approach. The proposed solution decouples the sensor’s data streams from the application-specific infrastructure and migrates them into a software-defined databus that can be configured to suit the end application’s demands. To further validate DGSS capabilities, a DER oriented use case has been developed.

Sensor Dissemination Networks, Enhanced DER visibi↗
DOE OSTI2024

Real-Time Ethernet Interface for NSTX-U’s Thomson Scattering Diagnostic (2023)

Here, the multipoint Thomson scattering (MPTS) diagnostic system at the National Spherical Torus Experiment Upgrade (NSTX-U) facility is undergoing an upgrade to operate in real-time and interface with the plasma control system (PCS) for NSTX-U. Previous prototyping efforts have shown that spectral analysis and rapid calculations of electron temperature and density are possible on a real-time Linux machine when using up to a 100-Hz laser pulse repetition rate. A remaining challenge was transferring the real-time data to NSTX-U’s PCS, which utilizes the front panel data port (FPDP) protocol. The original proposed method was to convert the real-time data into analog values, but a new solution was developed to keep the output format digital by using an Ethernet controller with a field-programmable gate array (FPGA). This article focuses on a new input module that has been developed to accept incoming user datagram protocol (UDP) packets sent over Ethernet, convert into FPDP format, and integrate into the existing data stream under NSTX-U’s real-time framework.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2025

Proximal remote sensing: an essential tool for bridging the gap between high‐resolution ecosystem monitoring and global ecology

Summary A new proliferation of optical instruments that can be attached to towers over or within ecosystems, or ‘proximal’ remote sensing, enables a comprehensive characterization of terrestrial ecosystem structure, function, and fluxes of energy, water, and carbon. Proximal remote sensing can bridge the gap between individual plants, site‐level eddy‐covariance fluxes, and airborne and spaceborne remote sensing by providing continuous data at a high‐spatiotemporal resolution. Here, we review recent advances in proximal remote sensing for improving our mechanistic understanding of plant and ecosystem processes, model development, and validation of current and upcoming satellite missions. We provide current best practices for data availability and metadata for proximal remote sensing: spectral reflectance, solar‐induced fluorescence, thermal infrared radiation, microwave backscatter, and LiDAR. Our paper outlines the steps necessary for making these data streams more widespread, accessible, interoperable, and information‐rich, enabling us to address key ecological questions unanswerable from space‐based observations alone and, ultimately, to demonstrate the feasibility of these technologies to address critical questions in local and global ecology.

Plant Sciences↗
DOE OSTI2023

Approaches for Synthesis and Deployment of Controller Models on Automated Vehicles for Car-following in Mixed Autonomy

This paper describes the software design patterns and vehicle interfaces that were employed to transition vehicle controllers from simulation environments to open-road field experiments. The approach relies on a life cycle that utilizes model-based design and code generation, along with agile software development, and both software and hardware-in-the-loop testing, with additional safety margins. Autonomous designs should consider the dynamics of mixed autonomy in traffic to safely operate among humans. The software that provides a vehicle’s behavior intelligence is often developed through simulation, which may have a mismatch between dynamics, or as a result of a reinforcement learning workflow, which may be a black box with challenges to analyze. In each of these cases, it is important to have research interfaces that provide strongly typed data streams accessible to researchers who are not software experts while continuing to satisfy safety and liveness constraints. This paper describes how we design the hardware platform interfaces and software design process for a mixed autonomy traffic experiment with a leader-follower scenario. Controller synthesis for these vehicles requires clearly articulated vehicle interfaces and software design patterns for successful onboard deployment. Testing strategies for such controllers are also described before algorithms are transitioned to full-scale field experiments with safety operators for the vehicles. Testing strategies include software-in-the-loop simulation testing, hardware-in-the-loop simulation, ghost-car testing, and read-only testing in live traffic. With our approach, we were not only able to validate our controller synthesized in scripts and simulation, but also able to scale deployment to multiple vehicles.

Bhadani, Rahul↗
DOE OSTI2025

Development of Emergency Vehicle Preemption Strategies on Smart Corridors in a Digital Twin Environment

Emergency Response Vehicles (ERVs), such as firetrucks, ambulances, etc. operate with the purpose of saving lives and mitigating property damage. As such, ERV travel-time reductions may result in significant benefits to the community. A common strategy to improve travel times is Emergency Vehicle Preemption (EVP). EVP seeks to reduce ERV delays by providing the right-of-way to ERVs as they approach an intersection. This study proposes a new Dynamic Preemption strategy that determines the need for preemption prior to the ERV reaching the vicinity of the intersection, utilizing real-time data streams. This paper evaluates the effectiveness of some existing and proposed preempt control strategies using a digital twin testbed consisting of a series of signalized intersections on an urban arterial in Georgia. The best EVP strategy maximizes the improvement in ERV travel time while minimizing the adverse effect of preemption on the traffic in conflicting directions. Therefore, this study evaluates both the positive impact of EVP on the ERV as well as the adverse impact on the cross-street traffic. The study found that the potential exists for significant improvements in ERV travel time with the proposed Dynamic Preemption strategy, with minimal impact to the conflicting traffic. For the simulation corridor there was a 20% reduction in the ERV travel times with the implementation of the Dynamic Preemption strategy, compared to traditional EVP practices.

Roy, Somdut [Georgia Institute of Technology]↗
DOE OSTI2025

Accelerating Advanced Light Source Science Through Multi-Facility HPC Workflows

Synchrotron light sources support a wide array of techniques to investigate materials, often producing complex, high-volume data that challenge traditional workflows. At the Advanced Light Source (ALS), we developed infrastructure to move microtomography data over ESnet to ALCF and NERSC, where CPU- and GPU-based algorithms generate 3D reconstructed volumes of experimental samples. We employ two data movement and reconstruction models: real-time processing as data streams directly to NERSC compute nodes, and automated file transfer to NERSC and ALCF file systems. The streaming pipeline provides users with feedback in under ten seconds, while the file-based workflow produces high-quality reconstructions suitable for deeper analysis in 20-30 minutes. This infrastructure enables users to utilize HPC resources without direct access to backend systems. We plan to extend this architecture to more endstations, supporting our beamline scientists and users.

Abramov, David↗
DOE OSTI2020

Commercial Routing Assistance

The Commercial Routing Assistance (CRA) tool merges coordinated and vetted data streams, plots multiple automated or custom routing options, and visualizes the wide variety of state regulations and actions that a driver would encounter along a route. With CRA, commercial operators can plan vehicle movements across multiple states quickly, particularly during emergencies.

Klett, TimothyR.↗
DOE OSTI2022

DER Cybersecurity Detection and Response Suite

SAND2024-08475O The Distributed Energy Resource (DER) Cybersecurity Detection and Response Suite is a solution for distributed energy resource (DER) systems. The DER Security Orchestration, Automation, and Response (SOAR) solution that uses alerts from signature- and behavior-based Intrusion Detection Systems are intended to be deployed as bump-in-the-wire (BITW) devices in front of DER equipment. The fielded application would use multiple intrusion detection systems that report data to SOAR to respond to cyberattacks. The suite consists of two software components: • The proactive intrusion detection and mitigation system (PIDMS) secures grid-edge photovoltaic smart inverters and other equipment in distributed energy resource systems. It is a distributed BITW solution; cyber and physical data are automatically processed using network inspection tools and custom machine learning algorithms to detect abnormal events and correlate cyber-physical events. • The Security Orchestration, Automation, and Response for Distributed Energy Resources (SOAR4DER) application ingests data from several intrusion detection systems to quickly block attacks and revert DER systems to good states. Using a collection of intrusion detection system technologies on a BITW device, it incorporates physical and cyber data to detect abnormal and potential malicious behaviors. Multiple SOAR playbooks then use the intrusion detection system data streams to automatically defend the system. SOAR4DER system testing showed detection and response times under 30 seconds for all adversary reconnaissance, denial-of-service attacks, malicious Modbus commands, brute-force logins, and machine-in-the-middle attacks. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Johnson, Jay↗
DOE OSTI2021

A Geometry-Driven Longitudal Topic Model

A simple and scalable framework for longitudinal analysis of Twitter data is developed that combines latent topic models with computational geometric methods. Dimensionality reduction tools from computational geometry are applied to learn the intrinsic manifold on which the latent, temporal topics reside. Then shortest path distances on the manifold are used to link together these topics. The proposed framework permits visualization of the low-dimensional embedding which provides clear interpretation of the complex, high-dimensional trajectories that may exist among latent topics. Practical application of the proposed framework is demonstrated through its ability to capture and effectively visualize natural progression of latent COVID-19 related topics learned from Twitter data. Interpretability of the trajectories is achieved by comparing to real-world events. In addition, the framework permits study of spatial variation in Twitter behavior for learned topics. The analysis demonstrates that the proposed framework is able to capture granular-level impact of COVID-19 on public discussions. We end by arguing that Twitter data, when analyzed within the proposed framework, can serve as a valuable supplementary data stream for COVID-related studies.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

The Surface Atmosphere Integrated Field Laboratory (SAIL) Campaign

The science of mountainous hydrology spans the atmosphere through the bedrock and inherently crosses physical and disciplinary boundaries: land-atmosphere interactions in complex terrain enhance clouds and precipitation, while watersheds retain and release water over a large range of spatial and temporal scales. Limited observations in complex terrain challenge efforts to improve predictive models of the hydrology in the face of rapid changes. The Upper Colorado River exemplifies these challenges, especially with ongoing mismatches between precipitation, snowpack, and discharge. Consequently, the U.S. Department of Energy’s (DOE) Atmospheric Radiation Measurement (ARM) user facility has deployed an observatory to the East River Watershed near Crested Butte, Colorado between September 2021 and June 2023 to measure the main atmospheric drivers of water resources, including precipitation, clouds, winds, aerosols, radiation, temperature and humidity. This effort, called the Surface Atmosphere Integrated Field Laboratory (SAIL), is also working in tandem with DOE-sponsored surface and subsurface hydrologists and other federal, state, and local partners. SAIL data can be benchmarks for model development by producing a wide range of observational information on precipitation and its associated processes, including those processes that impact snowpack sublimation and redistribution, aerosol direct radiative effects in the atmosphere and in the snowpack, aerosol impacts on clouds and precipitation, and processes controlling surface fluxes of energy and mass. Preliminary data from SAIL’s first year showcase the rich information content in SAIL’s many data-streams and support testing hypotheses that will ultimately improve scientific understanding and predictability of Upper Colorado River hydrology in 2023 and beyond.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

HUMAN-ARTIFICIAL INTELLIGENCE TEAMING FOR THE U.S. NAVY: DEVELOPING A HOLISTIC RESEARCH ROADMAP

With the ever-increasing deluge of data and demand for warfighters to make decisions upon its analysis, U.S. defense strategy has prioritized the development of artificial intelligence (AI)/machine learning (ML) systems that can analyze multi-source data streams and suggest courses of action. However, there is a history of systems that have failed to be adopted by the warfighter due not only to unsolved technical challenges, but also a lack of usability or contributions to mission effectiveness, perceived or otherwise. To avoid this, research into human-AI teaming shows promise for developing AI systems that work with frontline operators. A recent National Academies of Sciences, Engineering, and Medicine report (NASEM, 2022) presented 57 research objectives in this area; however, the U.S. Navy requires a more-focused set of priorities, as it is impossible to tackle every priority. A workshop involving 23 human factors scientists, computer scientists, and active-duty sailors was organized at the Naval Information Warfare Center Pacific, resulting in a set of five research priorities spanning near-, mid-, and far-term time frames. This panel will summarize the results of this workshop, with a focus on the big questions both going into this workshop and coming out of it. The panel participants come from government, academia, and industry, providing perspective from the different kinds of organizations required to accomplish these research goals.

Wong, Jason↗
DOE OSTI2023

Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction

Abstract Background West Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental USA. WNV occurrence has high spatiotemporal variation, and current approaches to targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement. Methods We performed a multi-model comparative assessment of probabilistic forecasts submitted by 15 teams for annual WNND cases in US counties for 2020 and assessed forecast accuracy, calibration, and discriminatory power. In the evaluation, we included forecasts produced by comparison models of varying complexity as benchmarks of forecast performance. We also used regression analysis to identify modeling approaches and contextual factors that were associated with forecast skill. Results Simple models based on historical WNND cases generally scored better than more complex models and combined higher discriminatory power with better calibration of uncertainty. Forecast skill improved across updated forecast submissions submitted during the 2020 season. Among models using additional data, inclusion of climate or human demographic data was associated with higher skill, while inclusion of mosquito or land use data was associated with lower skill. We also identified population size, extreme minimum winter temperature, and interannual variation in WNND cases as county-level characteristics associated with variation in forecast skill. Conclusions Historical WNND cases were strong predictors of future cases with minimal increase in skill achieved by models that included other factors. Although opportunities might exist to specifically improve predictions for areas with large populations and low or high winter temperatures, areas with high case-count variability are intrinsically more difficult to predict. Also, the prediction of outbreaks, which are outliers relative to typical case numbers, remains difficult. Further improvements to prediction could be obtained with improved calibration of forecast uncertainty and access to real-time data streams (e.g. current weather and preliminary human cases). Graphical Abstract

54 ENVIRONMENTAL SCIENCES↗