Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

KAZRARSCL-CLOUDSAT Value Added Product Data Stream

The KAZRARSCL-CLOUDSAT Value-Added Product (VAP) is based on the KAZR-ARSCL VAP, which provides cloud boundaries and best-estimate time-height fields of radar moments. The KAZRARSCL-CLOUDSAT VAP applies a statistically-derived calibration offset to reflectivity fields in order to align them with observations from the spaceborne CloudSat Cloud Profiling Radar. For details on the offset values applied, please refer to "Kollias, P., Puigdomènech Treserras, B., and Protat, A.: Calibration of the 2007–2017 record of ARM Cloud Radar Observations using CloudSat, Atmos. Meas. Tech. Discuss., https://doi.org/10.5194/amt-2019-34, 2019."

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

ML Classifier Fusion for Three Data Streams with Quality Inversely Proportional to Time Resolution

We consider a monitoring scenario of phenomenon using three different streams of measurements whose quality is proportional to their constant inter-arrival times. Each measurement of a stream needs to be binary-classified to reflect the state of interest of the phenomenon. A set of classifiers is separately trained and fused for each stream at its time resolution using measurements collected under known states. We present a machine learning method to fuse the outputs of these fusers to provide a final classification at the finest time resolution. We show that this fused-fusers method provides decisions with likely superior classification probability compared to the best individual classifiers and fused-classifiers. We derive generalization equations that guarantee a superior classification probability of fused-fusers with a confidence probability specified by the classifiers’ generalization equations. We apply these results to study a practical problem of classifying Pu/Np target dissolution events at a radiochemical processing facility using gamma spectral measurements of effluent flows.

Rao, Nageswara↗
DOE OSTI2023

Toward Statistical Real-Time Power Fault Detection

We propose statistical fault detection methodology based on high-frequency data streams that are becoming available in modern power grids. Our approach can be treated as an online (sequential) change point monitoring methodology. However, due to the mostly unexplored and very nonstandard structure of high-frequency power grid streaming data, substantial new statistical development is required to make this methodology practically applicable. The paper includes development of scalar detectors based on multichannel data streams, determination of data-driven alarm thresholds and investigation of the performance and robustness of the new tools. Due to a reasonably large database of faults, we can calculate frequencies of false and correct fault signals, and recommend implementations that optimize these empirical success rates.

bolted faults↗
DOE OSTI2023

A Three-Tier Incremental Approach for Development of Smart Corridor Digital Twins

Development of an arterial smart corridor digital twin requires the integration of real time data streams, data storage, and simulation model execution. This process may be complex, time consuming, and susceptible to errors. To aid in smart corridor digital twin development this paper seeks to provide a framework, best practices, and development guidance. As such, a three-tier incremental approach to smart corridor digital twin development is presented. The paper highlights practical issues in digital twin construction along with key data challenges based on experiences from the development of digital twins for two large Smart Corridor deployments, one in Chattanooga, Tennessee, and the other in Atlanta, Georgia. The presented three-tier incremental approach includes: 1) development of a prepopulated offline simulation, 2) development of a pseudo digital twin that is driven by archived data, and 3) integration of real time data streams to create the online digital twin model. The three-tiered approach facilitates conducting multiple trials and scenarios with increasing complexity, allowing for incremental error processing and updating of the digital twin.

Saroj, Abhilasha↗
DOE OSTI2021

Evolutionary NAS in Light of Model Stability for Accurate Continual Learning

Continual learning, the capability to learn new knowledge from streaming data without forgetting the previous knowledge, is a critical requirement for dynamic learning systems, especially for emerging edge devices such as self-driving cars and drones. However, continual learning is still facing the catastrophic forgetting problem. Previous work illustrate that model performance on continual learning is not only related to the learning algorithms but also strongly dependent on the inherited model, i.e., the model where continual learning starts. The better stability of the inherited model, the less catastrophic forgetting and thus, the inherited model should be elaborately selected. Inspired by this finding, we develop an evolutionary neural architecture search (ENAS) algorithm that emphasizes the Stability of the inherited model, namely ENAS-S. ENAS-S aims to find optimal architectures for accurate continual learning on edge devices. On CIFAR-10 and CIFAR-100, we present that ENAS-S achieves competitive architectures with lower catastrophic forgetting and smaller model size when learning from a data stream, as compared with handcrafted DNNs.

Du, Xiaocong↗
DOE OSTI2026

Characterizing the GD-1 Stream with DESI DR2 Data: Thin Stream and Hot Cocoon

GD-1 is among the longest, coldest stellar streams in the Milky Way, making it an ideal target for probing dark matter substructure through dynamical heating. We present a catalog of 608 spectroscopically confirmed GD-1 members from the first three years of Dark Energy Spectroscopic Instrument (DESI) observations. This constitutes the largest homogeneous spectroscopic sample of GD-1, doubling the number of members previously available only through heterogeneous compilations combining multiple surveys with different systematics. Using these data, we derive updated stream tracks in sky position, proper motion, and radial velocity that extend over $100^\circ$ of the stream. We apply a Gaussian mixture model to decompose the stream into a dynamically cold thin component ($σ_V = 2.49\pm 0.28$ km s$^{-1}$, width $= 0.23\pm0.01^\circ$) and a kinematically hot cocoon ($σ_V = 6.13\pm0.75$ km s$^{-1}$, width $= 2.18\pm0.17^\circ$). The cocoon contains $\sim30\%$ of members and its velocity dispersion is consistent with $\sim11$ Gyr of heating by cold dark matter subhalos. We also detect a large proper motion dispersion ($41.36\pm4.98$ km s$^{-1}$) along the stream direction in the cocoon component. This feature indicates a significant line-of-sight distance spread in the cocoon, and its origin will be further explored in a forthcoming paper. These measurements demonstrate the power of DESI spectroscopy for characterizing the multi-component phase-space structure of stellar streams and constraining small-scale dark matter substructure.

Jarvis, Emma [Toronto U.] (ORCID:0009000656127336)↗
DOE OSTI2020

SAMPA Based Streaming Readout Data Acquisition Prototype

We have assembled a small-scale streaming data acquisition system based on the SAMPA front-end ASIC. The 32-channel SAMPA chip was designed for the high-luminosity upgrade of the ALICE Time Projection Chamber (TPC) detector at the CERN Large Hadron Collider. The goals of the prototype system are to determine if the SAMPA chip is appropriate for use in detector systems at Jefferson Lab, and to gain experience with the hardware and software required to deploy streaming data acquisition systems in nuclear physics experiments. The 800 channel system is composed of components used in the ALICE TPC data acquisition upgrade. Five front-end cards (FEC) support five SAMPA chips each. SAMPA data streams on an FEC are concentrated into two high-speed (4.48 Gb/s) serial data streams by a pair Gigabit Transceiver ASICs (GBTx). These ten streams (44.8 Gb/s) are transmitted from the FECs over fibers to a PCIe based readout unit. The FPGA engine on the readout unit compresses data for transmission to a server via 100 Gb ethernet. Components on the FECs are radiation tolerant. High data rates can be handled. The system is by design scalable and thus provides a functional prototype for high-rate streaming readout at Jefferson Lab and the future Electron Ion Collider. We have made fundamental measurements (noise, linearity, time resolution) on the SAMPA ASIC. We have also coupled the readout system to a small Gas Electron Multiplier (GEM) detector and have studied its response to cosmic rays. A beam test of the system is planned.

ABBOTT, David↗
DOE OSTI2022

Electronic device and method for compressing video data

A computer-implemented method for compressing video data comprises receiving a sequence of video data values, each video data value being a digital value from a successive one of a plurality of pixels that form a video sensor, the sequence of video data values resulting from successive frames of video captured by the video sensor; extracting the video data values for each pixel in turn to create a plurality of pixel data streams, each pixel data stream including the video data value for each frame of captured video for the pixel; and applying data compression to each pixel data stream to create compressed data for each pixel data stream.

Tohlen, Michael Aaron↗
DOE OSTI2023

Electronic device and method for compressing video data

A computer-implemented method for compressing video data comprises receiving a sequence of video data values, each video data value being a digital value from a successive one of a plurality of pixels that form a video sensor, the sequence of video data values resulting from successive frames of video captured by the video sensor; extracting the video data values for each pixel in turn to create a plurality of pixel data streams, each pixel data stream including the video data value for each frame of captured video for the pixel; and applying data compression to each pixel data stream to create compressed data for each pixel data stream.

Tohlen, Michael Aaron↗
DOE OSTI2021

Transmitter with fully re-assignable segments for reconfigurable FFE taps

Methods and systems of performing feed forward equalization (FFE) on data streams are described. A circuitry may generate staggered data streams from data streams of an input signal. The staggered data streams may include data in staggered unit intervals. The circuitry may include a plurality of segments. A segment may define a specific unit interval to carve the staggered data streams into one unit interval pulses positioned at the specific unit interval. The specific unit interval to carve the staggered data streams may indicate an assignment of the segment as one of a FFE pre tap, a FFE main tap, and a FFE post tap. The plurality of segments may be assigned to different FFE taps based on different clock signal selection defining different unit intervals to perform the carving. The plurality of segments may output respective one unit interval pulses to reproduce the input signal.

Toprak-Deniz, Zeynep↗
DOE OSTI2023

Data transmission framing

Techniques for framing data in various data transmission contexts are described. A data framing technique may include a transmitter sending a data stream including repeating bits in alternating forward and reverse order. A receiver of the data stream may fold the data stream, and correlate portions of the folded data stream for purposes of validating the data stream and/or identifying an ID in the data stream. In at least some instances, once the receiver validates the data stream, the receiver may accept payload accompanying the data stream.

Palmer, David↗
DOE OSTI2021

Fractional delay filter for a digital signal processing system

A processing element for implementation in a digital signal processing system is provided. The processing element is configured to receive a first data stream comprising a plurality of digital values where each value represents a sample of an analog signal. The processing element is further configured to receive a second data stream comprising a series of digital values where each value represents a sample of the analog signal. The processing element is configured to filter the first data stream via a first Farrow-structured fractional delay (FD) filter and output a filtered first data stream; filter the second data stream via a second Farrow-structured FD filter and output a filtered second data stream; and temporarily store values from the second data stream and output the stored values to the first Farrow-structured FD filter so that the stored values can be used to filter the first data stream.

Stanley, Dennis L.↗
DOE OSTI2023

FUN-BioCROP model with litter decomposition parameters derived from the LIDET dataset

This repository contains the code and data necessary to run the FUN-BioCROP (Fixation and Uptake of Nitrogen-Bioenergy Carbon, Rhizosphere, Organisms, and Protection) model with litter decomposition parameters derived from a modified Monte Carlo simulation that used the Long-term Intersite Decomposition Experiment Team dataset. Related publication:Juice, S.M., Ridgeway, J.R., Hartman, M.D., Parton, W.J., Berardi, D.M., Sulman, B.N., Allen, K.E., & Brzostek, E.R. Reparameterizing litter decomposition using a simplified Monte Carlo method improves litter decay simulated by a microbial model and alters bioenergy soil carbon estimates. Description of Files: FUNBioCROP_LIDET Study.Rmd R code with FUN-BioCROP model that can be run with 10 different sets of parameters for litter decomposition (Baseline, LIDET, or eight other best parameter sets identified in the modified Monte Carlo simulation. CORPSE Functions_Bioenergy_V2.R Code with CORPSE model functions, called by FUNBioCROP_LIDET Study.Rmd Model Input Data: bulk.csv, bulk_till.csv, rhizo.csv, rhizo_till.csv, litter.csv Initial C and N (kg C or N/m2) pool values for each soil compartment, final values from spin up. All five files have the same columns: (Column - Description - Units) uFastC - Unprotected fast decomposing carbon - kg carbon/m2 uSlowC - Unprotected slow decomposing carbon - kg carbon/m2 uNecroC - Unprotected necromass carbon - kg carbon/m2 pFastC - Protected fast decomposing carbon - kg carbon/m2 pSlowC - Protected slow decomposing carbon - kg carbon/m2 pNecroC - Protected necromass carbon - kg carbon/m2 livingMicrobeC - Carbon in living microbial biomass - kg carbon/m2 uFastN - Unprotected fast decomposing nitrogen - kg nitrogen/m2 uSlowN - Unprotected slow decomposing nitrogen - kg nitrogen/m2 uNecroN - Unprotected necromass nitrogen - kg nitrogen/m2 pFastN - Protected fast decomposing nitrogen - kg nitrogen/m2 pSlowN - Protected slow decomposing nitrogen - kg nitrogen/m2 pNecroN - Protected necromass nitrogen - kg nitrogen/m2 inorganicN - Inorganic nitrogen - kg nitrogen/m2 CO2 - Carbon in carbon dioxide - kg carbon/m2 livingMicrobeN - Nitrogen in living microbial biomass - kg nitrogen/m2 Model Input Data: FluxTower_AvgSoilT.csv: Average daily soil temperature (oC) at 10 cm depth at University of Illinois Urbana-Champaign (UIUC) Energy Farm flux tower from 7/2008-3/2016. (One year of averaged data) Model Input Data: FluxTower_AvgSoilVWC.csv: Average daily soil volumetric water content (VWC) at 10 cm depth at UIUC Energy Farm flux tower from 7/2008-3/2016. (One year of averaged data) Model Input Data: input_CCS_LIDET Study.csv: This file has daily data to run FUN-BioCROP (Column - Description - Units): yr - calendar year - year doy - day of year (1 to 365) (no leap year) - day anpp - aboveground NPP (DayCent) - kg C/m2/day bnpp - belowground NPP (DayCent) - kg C/m2/day aglivc - live aboveground biomass carbon (DayCent) - kg C/m2 bglivcj - live juvenile fine root biomass carbon (DayCent) - kg C/m2 bglivcm - live mature fine root biomass carbon (DayCent) - kg C/m2 aglivn - live aboveground biomass nitrogen (DayCent) - kg N/m2 bglivnj - live juvenile fine root biomass nitrogen (DayCent) - kg N/m2 bglivnm - live mature fine root biomass nitrogen (DayCent) - kg N/m2 nyr - simulation year - year cult - indicates a cultivation event (0 or 1) crop - indicates a new crop (0 or 1) fert - indicates a fertilizer event (0 or 1) frst - indicates the first day of the growing season (0 or 1) harv - indicates a harvest event (0 or 1) last - indicates the end of the growing season (0 or 1) croptype - crop type (0=none; 1=alfalfa; 2=corn; 3=grass clover pasture; 4=soybean; 5=wheat) cropsrl - crop specific root length - mm/g root cultrhizmix - fraction of rhizosphere mixed with bulk soil during cultivation (0.0-1.0) - fraction cultlitmix - fraction of litter mixed with bulk soil during cultivation (0.0-1.0) - fraction harvremov - fraction of above ground biomass removed during harvest (0.0-1.0) - fraction fertamt - fertilization amount - g N/m2 lifehist - plant life history (0 = annual, 1 = perennial) froot_turnover_c - amount of C in fine root turnover - kg C/m2 froot_turnover_n - amount of N in fine root turnover - kg N/m2 agrd_turnover_c - amount of C in aboveground biomass turnover - kg C/m2 agrd_turnover_n - amount of N in aboveground biomass turnover - kg N/m2 leaf_litter_fastfrac - Fast decomposing fraction of leaf litter (0.0-1.0) - fraction root_litter_fastfrac - Fast decomposing fraction of root litter (0.0-1.0) - fraction root_diameter - root diameter - mm root_length - root length - mm root/m2 rhizo_frac - fraction of total soil volume that is rhizosphere (0.0 - 1.0) - fraction date - date in format YYYY-MM-DD Instructions: Save the model code ("FUN-BioCROP_LIDET Study.Rmd") and accompanything files (data streams and CORPSE function code) in the same folder. In model code "Chunk 3: Load CORPSE Data Streams" set the working directory (setwd) to the folder with the files saved in step #1. In "Chunk 5: Define LIDET parameter sets" select the litter decomposition parameter set to be used in the run, and comment out all other sets. If changing any parameter values, edit them in "Chunk 6: Load parameters." Run all chunks up to and including "Chunk 10: Prepare Data for Export." In "Chunk 11: Export Output Data" edit data frames for export and filenames, as necessary. "Chunk 12: Graph Total Soil C" makes a figure of C remaining over the model run period. Description of each model chunk (in file FUN-BioCROP_LIDET Study.Rmd): Chunk 1: Remove all functions, clear memory. Removes all functions from R environment, clears the memory. Chunk 2: Load Packages. Loads packages necessary to run the code. Chunk 3: Load CORPSE Data Streams. Sets the working directory and loads the data files necessary to run CORPSE. Chunk 4: Load CORPSE Functions. Loads the R script with CORPSE functions from the working directory, "CORPSE Functions_Bioenergy_V2.R". Chunk 5: Define LIDET parameter sets. Has ten different parameter sets for litter decomposition tested in this study: Baseline parameters, LIDET parameters, and the other 8 best performing parameter sets identified in the modified Monte Carlo. To run the model, all but one parameter set must be commented out. Chunk 6: Load Parameters. Loads all fixed parameters to run the model. Data frame with definitions of parameters is in the CORPSE function script "CORPSE Functions_Bioenergy_V2.R" Chunk 7: Prepare Data Streams. Takes data streams loaded in Chunk 3 and puts them in the format necessary to run the model. The model is coded to run at least two sites at a time, so if only one site is being run it must be run in duplicate. Individual data tables of daily values are created in this chunk from the input data file. Chunk 8: Set Initial Conditions. Creates data tables of soil C and N pools for each soil compartment (rhizo_till, rhizo, bulk_till, bulk, litter) and loads initial values into the data tables. Creates lists for each soil compartment to hold model output. Chunk 9: Load FUN Data and Set Up Matrices. Uses DayCent data to calculate FUN input data: root and leaf N demand, total N demand, plant CN, leaf N available for retranslocation, and litter production. Creates matrices for FUN model outputs. Chunk 10: Run Model. Runs the model. Chunk 11: Prepare Data for Export. Combines data from each day saved as lists into data frames for each soil compartment. Adds values from all soil compartments together to calculate total soil values, creates separate data frames for each soil C and N pool (e.g., protected slow C) for the total soil value. Adds different C and N pools together to calculate total soil C and N for all layers. Creates data frame of ratio of protected to unprotected SOC. Organizes FUN data for export. Chunk 12: Export Results. Exports CSV files of model results to the working directory. Chunk 13: Graph Total Soil C. Makes figure of C remaining over time. Related Links: Original FUN-BioCROP model: https://github.com/BrzostekEcologyLab/FUN-BioCROP LIDET dataset: https://andlter.forestry.oregonstate.edu/data/abstract.aspx?dbcode=TD023

Juice, Stephanie↗