SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

All sources NASA NTRS DOE OSTI

NASA NTRS1983

Spacelab high density digital recorders

The design and performance of the high-density digital recorder (HDDR) developed for use at the NASA centers (KSC, JSC, and GSFC) and at the JPL to store and retrieve 50-Mb/s PCM data streams from the Spacelab experiments are reported. The recording reproduction, and transport requirements are reviewed; and the design solutions adopted in the final version of the HDDR are described, incuding three-position-modulation and Y-phase encoding, microprocessor-controlled automatic bit synchronization and equalization, cyclic-redundancy-check error detection and correction, clock regeneration, data and clock variations, tape-speed control, and EEE-488 remote control. Reliable performance, with bit error rates 1 in 10 to the 10th forward and 1 in 10 to the 9th reverse or better and packing density up to 50 percent greater than that obtainable using conventional codes, is reported after 1.5 years of service.

Blais, R. A.↗

NASA NTRSDate not supplied

Data Science and the Knowledge Discovery Adventure

This talk will cover the important steps involved in the data science and knowledge discovery process: • Initial fact gathering (interview domain experts, review reports, articles, state-of-the-art) • Identify the problem (prediction, classification, statistical analysis, etc.) • Survey supporting data sources • Understand the data (numerical, categorical, text, sampling rate, data quality issues, etc.) • Selecting relevant features and sources • Acquire the data (set up agreements with the data stewards, APIs to download, etc.) • Merge data sources (temporal, spatial, common key, other ontologies...) • Feature Engineering (non linear domain knowledge or physics-based relationships) • Build data processing pipeline (may need to tap into data stream, develop parallel processing algorithm, federated learning etc.) • Build model and test (tune hyper-parameters, cross validation.) • Analyze/Validate results (do the results make sense. Does it answer the original question). • Deploy/Publish (Monitor and assess benefits)

Data science↗

DOE OSTI2022

Online Distribution System State Estimation via Stochastic Gradient Algorithm

Distribution network operation is becoming more challenging because of the growing integration of intermittent and volatile distributed energy resources (DERs). This motivates the development of new distribution system state estimation (DSSE) paradigms that can operate at fast timescale based on real-time data stream of asynchronous measurements enabled by modern information and communications technology. To solve the real-time DSSE with asynchronous measurements effectively and accurately, this paper formulates a weighted least squares DSSE problem and proposes an online stochastic gradient algorithm to solve it. The performance of the proposed scheme is analytically guaranteed and is numerically corroborated with realistic data on IEEE 123-bus feeder.

distribution system state estimation↗

NASA NTRS1990

A burst compression and expansion technique for variable-rate users in satellite-switched TDMA networks

A burst compression and expansion technique is described for asynchronously interconnecting variable-data-rate users with cost-efficient ground terminals in a satellite-switched, time-division-multiple-access (SS/TDMA) network. Compression and expansion buffers in each ground terminal convert between lower rate, asynchronous, continuous-user data streams and higher-rate TDMA bursts synchronized with the satellite-switched timing. The technique described uses a first-in, first-out (FIFO) memory approach which enables the use of inexpensive clock sources by both the users and the ground terminals and obviates the need for elaborate user clock synchronization processes. A continous range of data rates from kilobits per second to that approaching the modulator burst rate (hundreds of megabits per second) can be accommodated. The technique was developed for use in the NASA Lewis Research Center System Integration, Test, and Evaluation (SITE) facility. Some key features of the technique have also been implemented in the gound terminals developed at NASA Lewis for use in on-orbit evaluation of the Advanced Communications Technology Satellite (ACTS) high burst rate (HBR) system.

Budinger, James M.↗

NASA NTRS1990

A burst compression and expansion technique for variable-rate users in satellite-switched TDMA networks

A burst compression and expansion technique is described for asynchronously interconnecting variable-data-rate users with cost-efficient ground terminals in a satellite-switched, time-division-multiple-access (SS/TDMA) network. Compression and expansion buffers in each ground terminal convert between lower rate, asynchronous, continuous-user data streams and higher-rate TDMA bursts synchronized with the satellite-switched timing. The technique described uses a first-in, first-out (FIFO) memory approach which enables the use of inexpensive clock sources by both the users and the ground terminals and obviates the need for elaborate user clock synchronization processes. A continuous range of data rates from kilobits per second to that approaching the modulator burst rate (hundreds of megabits per second) can be accommodated. The technique was developed for use in the NASA Lewis Research Center System Integration, Test, and Evaluation (SITE) facility. Some key features of the technique have also been implemented in the ground terminals developed at NASA Lewis for use in on-orbit evaluation of the Advanced Communications Technology Satellite (ACTS) high burst rate (HBR) system.

Budinger, James M.↗

NASA NTRS2006

System and method for forward error correction

A system and method are provided for transferring a packet across a data link. The packet may include a stream of data symbols which is delimited by one or more framing symbols. Corruptions of the framing symbol which result in valid data symbols may be mapped to invalid symbols. If it is desired to transfer one of the valid data symbols that has been mapped to an invalid symbol, the data symbol may be replaced with an unused symbol. At the receiving end, these unused symbols are replaced with the corresponding valid data symbols. The data stream of the packet may be encoded with forward error correction information to detect and correct errors in the data stream.

Cole, Robert M.↗

NASA NTRS2007

System and method for transferring data on a data link

Cole, Robert M.↗

DOE OSTI2021

Automatic Traffic Queue-End Identification using Location-Based Waze User Reports

Traffic queues, especially queues caused by non-recurrent events such as incidents, are unexpected to high-speed drivers approaching the end of queue (EOQ) and become safety concerns. Though the topic has been extensively studied, the identification of EOQ has been limited by the spatial-temporal resolution of traditional data sources. This study explores the potential of location-based crowdsourced data, specifically Waze user reports. It presents a dynamic clustering algorithm that can group the location-based reports in real time and identify the spatial-temporal extent of congestion as well as the EOQ. The algorithm is a spatial-temporal extension of the density-based spatial clustering of applications with noise (DBSCAN) algorithm for real-time streaming data with an adaptive threshold selection procedure. Here, the proposed method was tested with 34 traffic congestion cases in the Knoxville, Tennessee area of the United States. It is demonstrated that the algorithm can effectively detect spatial-temporal extent of congestion based on Waze report clusters and identify EOQ in real-time. The Waze report-based detection are compared to the detection based on roadside sensor data. The results are promising: The EOQ identification time of Waze is similar to the EOQ detection time of traffic sensor data, with only 1.1 min difference on average. In addition, Waze generates 1.9 EOQ detection points every mile, compared to 1.8 detection points generated by traffic sensor data, suggesting the two data sources are comparable in respect of reporting frequency. The results indicate that Waze is a valuable complementary source for EOQ detection where no traffic sensors are installed.

99 GENERAL AND MISCELLANEOUS↗

DOE OSTI2021

X-composer: enabling cross-environments in-situ workflows between HPC and cloud

As large-scale scientific simulations and big data analyses become more popular, it is increasingly more expensive to store huge amounts of raw simulation results to perform post-analysis. To minimize the expensive data I/O, "in-situ" analysis is a promising approach, where data analysis applications analyze the simulation generated data on the fly without storing it first. However, it is challenging to organize, transform, and transport data at scales between two semantically different ecosystems due to the distinct software and hardware difference. To tackle these challenges, we design and implement the X-Composer framework. X-Composer connects cross-ecosystem applications to form an "in-situ" scientific workflow, and provides a unified approach and recipe for supporting such hybrid in-situ workflows on distributed heterogeneous resources. X-Composer reorganizes simulation data as continuous data streams and feeds them seamlessly into the Cloud-based stream processing services to minimize I/O overheads. For evaluation, we use X-Composer to set up and execute a cross-ecosystem workflow, which consists of a parallel Computational Fluid Dynamics simulation running on HPC, and a distributed Dynamic Mode Decomposition analysis application running on Cloud. Our experimental results show that X-Composer can seamlessly couple HPC and Big Data jobs in their own native environments, achieve good scalability, and provide high-fidelity analytics for ongoing simulations in real-time.

Wang, Dali↗

NASA NTRS2013

Sleuthing the MSL EDL Performance from an X Band Carrier Perspective

During the Entry, Descent, and Landing (EDL) of NASA's Mars Science Laboratory (MSL), or Curiosity, rover to Gale Crater on Mars on August 6, 2012 UTC, the rover transmitted an X-band signal composed of carrier and tone frequencies and a UHF signal modulated with an 8kbps data stream. During EDL, the spacecraft's orientation is determined by its guidance and mechanical subsystems to ensure that the vehicle land safely at its destination. Although orientation to maximize telecom performance is not possible, antennas are especially designed and mounted to provide the best possible line of sight to Earth and to the Mars orbiters supporting MSL's landing. The tones and data transmitted over these links are selected carefully to reflect the most essential parameters of the vehicle's state and the performance of the EDL subsystems for post-EDL reconstruction should no further data transmission from the vehicle be possible. This paper addresses the configuration of the X band receive system used at NASA / JPL's Deep Space Network (DSN) to capture the signal spectrum of MSL's X band carrier and tone signal, examines the MSL vehicle state information obtained from the X band carrier signal only and contrasts the Doppler-derived information against the post-EDL known vehicle state. The paper begins with a description of the MSL EDL sequence of events and discusses the impact of the EDL maneuvers such as guided entry, parachute deploy, and powered descent on the frequency observables expected at the DSN. The range of Doppler dynamics possible is derived from extensive 6 Degrees-Of-Freedom (6 DOF) vehicle state calculations performed by MSL's EDL simulation team. The configuration of the DSN's receive system, using the Radio Science Receivers (RSR) to perform open-loop recording for both for nominal and off-nominal EDL scenarios, is detailed. Expected signal carrier power-to-noise levels during EDL are shown and their impact on signal detection is considered. Particular attention is given to the selection of the appropriate RSR processing bandwidths and to its configuration for real-time signal detection. The X-band carrier frequency obtained through post-processing of the open-loop recorded spectrum is given. Detection of spacecraft status and completion of key vehicle events through their Doppler signature is discussed and illustrated. This Doppler-derived information is compared against the very accurate vehicle data obtained post-EDL via MSL's UHF radio subsystem. The paper concludes with a discussion on the advantages and disadvantages of transmitting the X-band carrier and tone signal in the general context of EDL communications and lessons learned for future missions with EDL sequences are given

Mars Science Laboratory (MSL)↗

DOE OSTI2021

Misuse Detection for a Generalized SFR Test Reactor

Sodium-cooled Fast Reactors (SFRs) present unique challenges for international safeguards. SFRs possess neutron physics characteristics that if configured appropriately could produce more fissile material than consumed. An adversary state may choose to build an SFR, justified by a lack of domestic natural uranium and limited access or interest to procuring uranium from international markets. Once constructed, the state may choose to misuse the SFR for the purpose of diverting fissile plutonium from declared operation. This work shows that a demonstration SFR does not need to be configured as a plutonium breeder to create one Significant Quantity (SQ) of plutonium in a short amount of time (e.g., one to few years). However, such an extreme case of misuse would change the core reactivity in such a way as to be easily indicated by deviations of control rod position compared to declared operation. In this work a contrived SFR demonstration reactor was modeled for the purpose of exploring proliferation scenarios and how such misuse could be detected using the SFR's Reactor Data Acquisition System (RDAS). Typically, the International Atomic Energy Agency (IAEA) does not have access to the control rod position, power, thermal, pressure sensing and indicating systems of nuclear power plants. However, this work shows that such data streams can be compared against a parallel detailed simulation model (a Digital Twin) to detect possible misuse.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

NASA NTRS1994

Multifractal characterizations of nonstationarity and intermittency in geophysical fields: Observed, retrieved, or simulated

Geophysical data rarely show any smoothness at any scale, and this often makes comparison with theoretical model output difficult. However, highly fluctuating signals and fractal structures are typical of open dissipative systems with nonlinear dynamics, the focus of most geophysical research. High levels of variability are excited over a large range of scales by the combined actions of external forcing and internal instability. At very small scales we expect geophysical fields to be smooth, but these are rarely resolved with available instrumentation or simulation tools; nondifferentiable and even discontinuous models are therefore in order. We need methods of statistically analyzing geophysical data, whether measured in situ, remotely sensed or even generated by a computer model, that are adapted to these characteristics. An important preliminary task is to define statistically stationary features in generally nonstationary signals. We first discuss a simple criterion for stationarity in finite data streams that exhibit power law energy spectra and then, guided by developments in turbulence studies, we advocate the use of two ways of analyzing the scale dependence of statistical information: singular measures and qth order structure functions. In nonstationary situations, the approach based on singular measures seeks power law behavior in integrals over all possible scales of a nonnegative stationary field derived from the data, leading to a characterization of the intermittency in this (gradient-related) field. In contrast, the approach based on structure functions uses the signal itself, seeking power laws for the statistical moments of absolute increments over arbitrarily large scales, leading to a characterization of the prevailing nonstationarity in both quantitative and qualitative terms. We explain graphically, step by step, both multifractal statistics which are largely complementary to each other. The geometrical manifestations of nonstationarity and intermittency, 'roughness' and 'sparseness', respectively, are illustrated and the associated analytical (differentiability and continuity) properties are discussed. As an example, the two techniques are applied to a series of recent measurements of liquid water distributions inside marine stratocumulus decks; these are found to be multifractal over scales ranging from approximately 60 m to approximately 60 km. Finally, we define the 'mean multifractal plane' and show it to be a simple yet comprehensive tool with many applications including data intercomparison, (dynamical or stochastic) model and retrieval validations.

Davis, Anthony↗

DOE OSTI2024

Time series methods for the analysis of soundscapes and other cyclical ecological data

Biodiversity monitoring has entered an era of ‘big data’, exemplified by a near-continuous collection of sounds, images, chemical and other signals from organisms in diverse ecosystems. Such data streams have the potential to help identify new threats, assess the effectiveness of conservation interventions, as well as generate new ecological insights. However, appropriate analytical methods are often still missing, particularly with respect to characterizing cyclical temporal patterns. Here, we present a framework for characterizing and analysing ecological responses that represent nonstationary, complex temporal patterns and demonstrate the value of using Fourier transforms to decorrelate continuous data points. In our example, we use a framework based on three approaches (spectral analysis, magnitude squared coherence, and principal component analysis) to characterize differences in tropical forest soundscapes within and across sites and seasons in Gabon. By reconstructing the underlying, cyclic behaviour of the soundscape for each site, we show how one can identify circadian patterns in acoustic activity. Soundscapes in the dry season had a complex diel cycle, requiring multiple harmonics to represent daily variation, while in the wet season there was less variance attributable to the daily cyclic patterns. Our framework can be applied to most continuous, or near-continuous ecological data collected at a fine temporal resolution, allowing ecologists to explore patterns of temporal autocorrelation at multiple levels for biologically meaningful trends. Such methods will become indispensable as biological big data are used to understand the impact of anthropogenic pressures on biodiversity and to inform efforts to mitigate them.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2024

On learning what to learn: Heterogeneous observations of dynamics and establishing possibly causal relations among them

Abstract Before we attempt to (approximately) learn a function between two sets of observables of a physical process, we must first decide what the inputs and outputs of the desired function are going to be. Here we demonstrate two distinct, data-driven ways of first deciding “the right quantities” to relate through such a function, and then proceeding to learn it. This is accomplished by first processing simultaneous heterogeneous data streams (ensembles of time series) from observations of a physical system: records of multiple observation processes of the system. We determine (i) what subsets of observables are common between the observation processes (and therefore observable from each other, relatable through a function); and (ii) what information is unrelated to these common observables, therefore particular to each observation process, and not contributing to the desired function. Any data-driven technique can subsequently be used to learn the input–output relation—from k-nearest neighbors and Geometric Harmonics to Gaussian Processes and Neural Networks. Two particular “twists” of the approach are discussed. The first has to do with the identifiability of particular quantities of interest from the measurements. We now construct mappings from a single set of observations from one process to entire level sets of measurements of the second process, consistent with this single set. The second attempts to relate our framework to a form of causality: if one of the observation processes measures “now,” while the second observation process measures “in the future,” the function to be learned among what is common across observation processes constitutes a dynamical model for the system evolution.

Sroczynski, David W.↗

DOE OSTI2022

Robust Event Classification Using Imperfect Real-world PMU Data

Here, this paper studies robust event classification using imperfect real-world phasor measurement unit (PMU) data. By analyzing the real-world PMU data, we find it is challenging to directly use this dataset for event classifiers due to the low data quality observed in PMU measurements and event logs. To address these challenges, we develop a novel machine learning framework for training robust event classifiers, which consists of three main steps: data preprocessing, fine-grained event data extraction, and feature engineering. Specifically, the data preprocessing step addresses the data quality issues of PMU measurements (e.g., bad data and missing data); in the fine-grained event data extraction step, a model-free event detection method is developed to accurately localize the events from the inaccurate event timestamps in the event logs; and the feature engineering step constructs the event features based on the patterns of different event types, in order to improve the performance and the interpretability of the event classifiers. Based on the proposed framework, we develop a workflow for event classification using the real-world PMU data streaming into the system in real time. Using the proposed framework, robust event classifiers can be efficiently trained based on many off-the-shelf lightweight machine learning models. Numerical experiments using the real-world dataset from the Western Interconnection of the U.S power transmission grid show that the event classifiers trained under the proposed framework can achieve high classification accuracy while being robust against low-quality data.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

DOE OSTI2022

Classification of Nuclear Reactor Operations Using Spatial Importance and Multisensor Networks

Distributed multisensor networks record multiple data streams that can be used as inputs to machine learning models designed to classify operations relevant to proliferation at nuclear reactors. The goal of this work is to demonstrate methods to assess the importance of each node (a single multisensor) and region (a group of proximate multisensors) to machine learning model performance in a reactor monitoring scenario. This, in turn, provides insight into model behavior, a critical requirement of data-driven applications in nuclear security. Using data collected at the High Flux Isotope Reactor at Oak Ridge National Laboratory via a network of Merlyn multisensors, two different models were trained to classify the reactor’s operational state: a hidden Markov model (HMM), which is simpler and more transparent, and a feed-forward neural network, which is less inherently interpretable. Traditional wrapper methods for feature importance were extended to identify nodes and regions in the multisensor network with strong positive and negative impacts on the classification problem. These spatial-importance algorithms were evaluated on the two different classifiers. The classification accuracy was then improved relative to baseline models via feature selection from 0.583 to 0.839 and from 0.811 ± 0.005 to 0.884 ± 0.004 for the HMM and feed-forward neural network, respectively. While some differences in node and region importance were observed when using different classifiers and wrapper methods, the nodes near the facility’s cooling tower were consistently identified as important—a conclusion further supported by studies on feature importance in decision trees. Node and region importance methods are model-agnostic, inform feature selection for improved model performance, and can provide insight into opaque classification models in the nuclear security domain.

Tibbetts, Jake↗

NASA NTRS2018

Land Surface Reflectances from Geostationary Sensors

GEONEX is a processing pipeline that produces a suite of satellite land surface products using data streams from the latest geostationary (GEO) sensors including the GOES016/ABI and the Himawari-8/AHI. The suite, created collaboratively by scientists from NASA and NOAA, includes top-of-atmosphere (TOA) reflectances, land surface reflectances (LSRs), vegetation indices, LAI/fPAR, and other downstream products. As a key component of the GEONEX product processing, we have adapted the Multi-Angle Implementation of Atmospheric Correction (MAIAC) algorithm to produce LSRs from the TOA data. Because the algorithm depends on building "stacks" of images, we first run internal geo-registration checks to ensure geo-spatial accuracy and consistency of the input (L1B) data before transferring them from the geostationary projection into a tile system in geographic grids. Scan-time is inferred from metadata and applied to calculate the sun-sensor angles for each grid cell. The MAIAC algorithm is run to detect clouds/shadows, estimate aerosol optical thickness (AOT), perform atmospheric corrections, and generate LSRs. We have processed 18-months (from 2016/04 onward) of AHI data over East Asia and Oceania at a 10-minute time step and 10-months (from 2018/01 onward) of ABI data over North and South Americas at a 15-minute time step. As a verification measure, we compare the GEONEX (AHI/ABI) surface reflectances with the standard MODIS products (MOD09GA) and the MODIS MAIAC products over pixels that have similar sun-view geometries. The results indicate general linear relationships between GEONEX and corresponding MODIS LSRs. In particular, the RMSEs between GEONEX and MOD09 data are comparable to those between MOD09 and MODIS MAIAC products, suggesting that the uncertainties of GEONEX LSRs fall into an acceptable range. However, direct comparisons of LSRs over pixels with different sun-view angles are not as straightforward and require more modeling efforts to correct the directional effects. Evaluation of such angular influences on the downstream products (e.g., vegetation indices) is also under investigation.

Geostationary satellite; Remote Sensing; Atmospher↗

NASA NTRS2023

Heterogeneity and Chemical Reactivity of the Remote Troposphere Defined By Aircraft Measurements - Corrected

The NASA Atmospheric Tomography (ATom) mission built a photochemical climatology of air parcels based on in situ measurements with the NASA DC-8 aircraft along objectively planned profiling transects through the middle of the Pacific and Atlantic oceans. In this paper we present and analyze a data set of 10 s (2 km) merged and gap-filled observations of the key reactive species driving the chemical budgets of O3 and CH4 (O3, CH4, CO, H2O, HCHO, H2O2, CH3OOH, C2H6, higher alkanes, alkenes, aromatics, NOx, HNO3, HNO4, peroxyacetyl nitrate, and other organic nitrates), consisting of 146 494 distinct air parcels from ATom deployments 1 through 4. Six models calculated the O3 and CH4 photochemical tendencies from this modeling data stream for ATom 1. We find that 80 %–90 % of the total reactivity lies in the top 50 % of the parcels and 25 %–35 % in the top 10 %, supporting previous model-only studies that tropospheric chemistry is driven by a fraction of all the air. Surprisingly, the probability densities of species and reactivities averaged on a model scale (100 km) differ only slightly from the 2 km ATom 10 s data, indicating that much of the heterogeneity in tropospheric chemistry can be captured with current global chemistry models. Comparing the ATom reactivities over the tropical oceans with climatological statistics from six global chemistry models, we find generally good agreement with the reactivity rates for O3 and CH4. Models distinctly underestimate O3 production below 2 km relative to the mid-troposphere, and this can be traced to lower NOx levels than observed. Attaching photochemical reactivities to measurements of chemical species allows for a richer, yet more constrained-to-what-matters, set of metrics for model evaluation.

Tropospheric Chemistry↗