Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 559 records · Page 31
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Developing Digital Twin Visualizations: A Methodology and Case Study on Chemical Separation Processing

As advances in digital engineering continue to push the technological boundaries, digital twin (DT) visualizations for diagnostics and safeguards advancement become much more feasible and practical. DTs generate large and complex data streams that require effective user interfaces to provide monitoring and diagnostic capabilities. Unfortunately, while these frameworks exist, there is not much research on the systematic documentation of human–computer interaction (HCI) for DT visualization. This work presents a dual-mode visualization methodology (two dimensional [2D] graphical user interface dashboard and 3D mixed reality) designed to support diagnostic tasks in DT systems and building on a validated framework and applying established HCI principles. The methodology is demonstrated through a case study of aqueous processing at Idaho National Laboratory, using experimental data from the chemical solvent extraction runs. Our interfaces display real-time alerts and monitoring to inform users of safeguards anomalies. The interfaces use immersive 3D mixed-reality visualization for further system and experiment investigation. This work demonstrates how the systematic application of HCI principles can inform DT visualization design for diagnostic and safeguards applications. While formal user evaluation studies remain as future work, this paper documents the systematic design methodology and demonstrates a proof-of-concept implementation.

3D visualization↗
DOE OSTI2024

Hydrology Outweighs Temperature in Driving Production and Export of Dissolved Carbon in a Snowy Mountain Catchment

Abstract Terrestrial production and export of dissolved organic and inorganic carbon (DOC and DIC) to streams depends on water flow and biogeochemical processes in and beneath soils. Yet, understanding of these processes in a rapidly changing climate is limited. Using the watershed‐scale reactive‐transport model BioRT‐HBV and stream data from a snow‐dominated catchment in the Rockies, we show deeper groundwater flow averaged about 20% of annual discharge, rising to ∼35% in drier years. DOC and DIC production and export peaked during snowmelt and wet years, driven more by hydrology than temperature. DOC was primarily produced in shallow soils (1.94 ± 1.45 gC/m 2 /year), stored via sorption, and flushed out during snowmelt. Some DOC was recharged to and further consumed in the deeper subsurface via respiration (−0.27 ± 0.02 gC/m 2 /year), therefore reducing concentrations in deeper groundwater and stream DOC concentrations at low discharge. Consequently, DOC was primarily exported from the shallow zone (1.62 ± 0.96 gC/m 2 /year, compared to 0.12 ± 0.02 gC/m 2 /year from the deeper zone). DIC was produced in both zones but at higher rates in shallow soils (1.34 ± 1.00 gC/m 2 /year) than in the deep subsurface (0.36 ± 0.02 gC/m 2 /year). Deep respiration elevated DIC concentrations in the deep zone and stream DIC concentrations at low discharge. In other words, deep respiration is responsible for the commonly‐observed increasing DOC concentrations (flushing) and decreasing DIC concentrations (dilution) with increasing discharge. DIC export from the shallow zone was ~66% of annual export but can drop to ∼53% in drier years. Numerical experiments suggest lower carbon production and export in a warmer, drier future, and a higher proportion from deeper flow and respiration processes. These results underscore the often‐overlooked but growing importance of deeper processes in a warming climate.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2016

Processing and Managing the Kepler Mission's Treasure Trove of Stellar and Exoplanet Data

The Kepler telescope launched into orbit in March 2009, initiating NASAs first mission to discover Earth-size planets orbiting Sun-like stars. Kepler simultaneously collected data for 160,000 target stars at a time over its four-year mission, identifying over 4700 planet candidates, 2300 confirmed or validated planets, and over 2100 eclipsing binaries. While Kepler was designed to discover exoplanets, the long term, ultra- high photometric precision measurements it achieved made it a premier observational facility for stellar astrophysics, especially in the field of asteroseismology, and for variable stars, such as RR Lyraes. The Kepler Science Operations Center (SOC) was developed at NASA Ames Research Center to process the data acquired by Kepler from pixel-level calibrations all the way to identifying transiting planet signatures and subjecting them to a suite of diagnostic tests to establish or break confidence in their planetary nature. Detecting small, rocky planets transiting Sun-like stars presents a variety of daunting challenges, from achieving an unprecedented photometric precision of 20 parts per million (ppm) on 6.5-hour timescales, supporting the science operations, management, processing, and repeated reprocessing of the accumulating data stream. This paper describes how the design of the SOC meets these varied challenges, discusses the architecture of the SOC and how the SOC pipeline is operated and is run on the NAS Pleiades supercomputer, and summarizes the most important pipeline features addressing the multiple computational, image and signal processing challenges posed by Kepler.

high performance computing↗
DOE OSTI2022

XRF-ROI Finder: Machine Learning to Guide Region-of-Interest Scanning for X-ray Fluorescence Microscopy

The ROI-finder software is being developed for use by several Microscopy Group beamlines at Argonne National Laboratory, including 2-ID microprobes and 9-ID-B Bionanoprobe which use multi-scale scanning fluorescence microscopy to acquire elemental maps (multi-modal image data). Microscopy experiments require scan of samples at a coarse resolution followed by ROI identification using feature detection based on domain expertise. Finer resolution scans are then conducted based on identified ROI. The decision-making process based on domain expertise will be difficult to perform for faster data rates and much larger sampling volumes anticipated after APS-U necessitating the need for the ROI-finder software. The ROI- finder detects regions of interest through a continuous learning process, starting with a unsupervised representation learning and improving its recommendations through supervised learning and an interactive tool for user annotation. The scope of ongoing development efforts includes the integration of image registration module to correlate optical and X-ray images, extraction of feature morphology as well as elemental signatures in the image space and incorporation of beamtime streaming data by the scanning probe via EPICS.

CHOWDHURY, M. ARSHAD ZAHANGIR↗
NASA NTRS1996

In search of cybernautics

This is a talk about the future of aviation in the information age. Ages come and go. Certainly the atomic age came and went, but the information age looks different. This talk reviews some recent experiments on navigation and control with the Global Positioning System. Vertical position accuracies within 1 foot have been demonstrated in the most recent experiments, and research emphases have shifted to issues of integrity, continuity, and availability. Inertial navigation systems (INS) contribute much to the reliability of GPS-based autoland systems. The GPS data stream can cease, and INS can still complete a precision landing from an altitude of 200 feet. The future of aviation looks like automatic airplanes communicating among each other to schedule ground assets and to avoid collisions and wake hazards. The business of the FAA will be to assure integrity of global navigation systems, to develop and maintain the software rules of the air, and to provide expert pilots to handle emergencies from the ground via radio control. The future of aviation is democratic and lends itself to personal airplanes. Some data analyses reveal that personal airplanes are just as efficient as large turbofan transports and just as fast over distances up to 1,000 miles, thanks to the decelerative influence of the hub and spoke system. Maybe by the year 2020, the airplane will rank with the automobile and computer as an agent of personal freedom.

Crow, Steven↗
DOE OSTI2021

In Situ Inference for Earth System Predictability

An understanding of future evolution in precipitation extremes is critical to numerous DOE mission questions. Extreme events are by nature short time-scale events that are difficult to diagnose in available model data. Accurate modeling of extreme events necessarily requires high spatial resolution at the storm scale locally. However, the environment in which storms grow is dependent on global, remote, processes. These complex spatiotemporal relationships are impossible to diagnose at resolutions required to accurately model storms responsible for extreme precipitation. At exascale, climate simulations will produce results at fine enough resolution to investigate these relationships. However, the resulting data from these simulations will be far too large to save for post-simulation analysis. We advocate for fitting statistical models inside the simulations as they run, a context known as in situ, which will facilitate scientific investigations using the full fine-scale data stream. Figure 1 shows an example of the type of model we could consider, a Bayesian hierarchical spatial regression model. Precipitation extremes at each grid cell are modeled using extreme value distributions. Since extremes are rare, fitting models to individual grid cells can result in high variance and poor estimates. Instead, the model can be made more robust by smoothing the parameters of the extreme value model across space. Additionally, the parameters themselves can be functionally linked to other variables elsewhere in the simulation. Thus, we can use the fine-scale data to build more robust models for extremes that link extreme behavior to other climate patterns.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

In Situ Inference for Earth System Predictability

Focal Area: Focal Area 3: Insight gleaned from complex simulated data using AI, big data analytics, and other advanced methods, including explainable AI and physics- or knowledge-guided AI. Science Challenge: An understanding of future evolution in precipitation extremes is critical to numerous DOE mission questions. Extreme events are by nature short time-scale events that are difficult to diagnose in available model data. Accurate modeling of extreme events necessarily requires high spatial resolution at the storm scale locally. However, the environment in which storms grow is dependent on global, remote, processes. These complex spatiotemporal relationships are impossible to diagnose at resolutions required to accurately model storms responsible for extreme precipitation. At exascale, climate simulations will produce results at fine enough resolution to investigate these relationships. However, the resulting data from these simulations will be far too large to save for post-simulation analysis. We advocate for fitting statistical models inside the simulations as they run, a context known as in situ, which will facilitate scientific investigations using the full fine-scale data stream. Figure 1 shows an example of the type of model we could consider, a Bayesian hierarchical spatial regression model. Precipitation extremes at each grid cell are modeled using extreme value distributions. Since extremes are rare, fitting models to individual grid cells can result in high variance and poor estimates. Instead, the model can be made more robust by smoothing the parameters of the extreme value model across space. Additionally, the parameters themselves can be functionally linked to other variables elsewhere in the simulation. Thus, we can use the fine-scale data to build more robust models for extremes that link extreme behavior to other climate patterns.

54 ENVIRONMENTAL SCIENCES↗
NASA NTRS2012

Entry, Descent, and Landing Communications for the 2011 Mars Science Laboratory

The Mars Science Laboratory (MSL), established as the most advanced rover to land on the surface of Mars to date, launched on November 26th, 2011 and arrived to the Martian Gale Crater during the night of August 5th, 2012 (PDT). MSL will investigate whether the landing region was ever suitable to support carbon-based life, and examine rocks, soil, and the atmosphere with a sophisticated suite of tools. This paper addresses the flight system requirement by which the vehicle transmitted indications of the following events using both X-band tones and UHF telemetry to allow identification of probable root causes should a mission anomaly have occurred: Heat-Rejection System (HRS) venting, completion of the cruise stage separation, turn to entry attitude, atmospheric deceleration, bank angle reversal commanded, parachute deployment, heatshield separation, radar ground acquisition, powered descent initiation, rover separation from the descent stage, and rover release. During Entry, Descent, and Landing (EDL), the flight system transmitted a UHF telemetry stream adequate to determine the state of the spacecraft (including the presence of faults) at 8 kbps initiating from cruise stage separation through at least one minute after positive indication of rover release on the surface of Mars. The flight system also transmitted X-band semaphore tones from Entry to Landing plus one minute although since MSL was occulted, as predicted, by Mars as seen from the Earth, Direct-To-Earth (DTE) communications were interrupted at approximately is approx. 5 min after Entry ( approximately 130 prior to Landing). The primary data return paths were through the Deep Space Network (DSN) for DTE and the existing Mars network of orbiting assets for UHF, which included the Mars Reconnaissance Orbiter (MRO), Mars Odyssey (ODY), and Mars Express (MEX) elements. These orbiters recorded the telemetry data stream and returned it back to Earth via the DSN. The paper also discusses the total power received during EDL and the robustness of the telecom design strategy used to ensure EDL communications coverage.

relay operations↗
NASA NTRS1993

SPNDL: A concept for a small satellite Doppler lidar wind sounder

Given current resources and the technical challenges in developing a full coverage space-based Doppler lidar wind measuring system such as LAWS (Laser Atmospheric Wind Sounder), it is not likely that the science community will have data streams with which to work before the end of this decade. Currently, a 'fast track' demonstration mission is being seriously considered by several U.S. agencies. Such a mission would have as its primary objectives the demonstration of coherent Doppler lidar technology in space and the delivery of wind observations for science algorithm evaluation and development. However, for such a mission to be achieved at modest costs and within a short time frame, deviations from the full system design are required. Simulation models have been developed over the last decade to aide in the design of Doppler lidar missions and to provide simulated data for use in wind computation algorithm development. SWA has used both models to examine some options that might be cost-effective for a demonstration mission. Over the past few months, Simpson Weather Associates has been studying SPNDL (Spinning Platform with a Non-rotating telescope Doppler Lidar), a new concept for Doppler lidar wind observations from space. Science and Technology Corporation has an interest in participating in an engineering and shuttle accommodation study for SPNDL.

Emmitt, G. D.↗
DOE OSTI2021

MODELING AND TESTING OF A NOVEL ULTRA-LOW TEMPERATURE SCO2 OPPOSING PISTON EXPANDER

The Flameless Pressurized Oxy-Combustion (FPO) cycle, a novel flexible fuel technology, is being developed into a large pilot. This effort seeks to complete the preliminary engineering and planning of a 25 MWth Pilot Plant that will demonstrate the technology for scale-up to a commercial unit. The technology, pioneered by ITEA at the 5 MWth scale, must be brought to a higher technology readiness level (TRL) to be viable at the commercial scale. The 25 MWth pilot cycle was optimized for cost and technology development and demonstration. Preliminary drawings, layouts, and plans were defined. Process flow diagrams were used to describe the pilot configuration in greater detail. A heat and mass balance with stream data was created. A master equipment list specified the operating conditions for major pieces of equipment within the pilot using this heat and mass balance. The 25 MWth FPO pilot is assessed for environmental performance. A test campaign is developed to assess the type of test and number of hours required for pilot demonstration. The environmental performance is compared against projected performance at the commercial scale. This project builds upon extensive evaluation of the techno-economic performance of the FPO technology already performed by ITEA. This includes system improvements, such as the addition of a turbo-expander to the flue gas stream. Some of the performance-enhancing components are not as well defined as others. The primary goal of this effort was to bring all of the core cycle components to the same level of design maturity. A techno-economic assessment (TEA) evaluated the FPO technology at the commercial scale in comparison to the NETL baseline cases. The reference plants were used in combination with proprietary equipment estimates to build a plant capital cost and cost of electricity evaluation. FPO performed better than the sub-bituminous post-combustion capture cases. Further preliminary estimates of improvements to the FPO cycle show even further gains when compared to conventional carbon capture methods.

01 COAL, LIGNITE, AND PEAT↗
DOE OSTI2021

DEVELOPMENT OF A 25 MWTH FLAMELESS PRESSURIZED OXY-COMBUSTION PILOT

The Flameless Pressurized Oxy-Combustion (FPO) cycle, a novel flexible fuel technology, is being developed into a large pilot. This effort seeks to complete the preliminary engineering and planning of a 25 MWth Pilot Plant that will demonstrate the technology for scale-up to a commercial unit. The technology, pioneered by ITEA at the 5 MWth scale, must be brought to a higher technology readiness level (TRL) to be viable at the commercial scale. The 25 MWth pilot cycle was optimized for cost and technology development and demonstration. Preliminary drawings, layouts, and plans were defined. Process flow diagrams were used to describe the pilot configuration in greater detail. A heat and mass balance with stream data was created. A master equipment list specified the operating conditions for major pieces of equipment within the pilot using this heat and mass balance. The 25 MWth FPO pilot is assessed for environmental performance. A test campaign is developed to assess the type of test and number of hours required for pilot demonstration. The environmental performance is compared against projected performance at the commercial scale. This project builds upon extensive evaluation of the techno-economic performance of the FPO technology already performed by ITEA. This includes system improvements, such as the addition of a turbo-expander to the flue gas stream. Some of the performance-enhancing components are not as well defined as others. The primary goal of this effort was to bring all of the core cycle components to the same level of design maturity. A techno-economic assessment (TEA) evaluated the FPO technology at the commercial scale in comparison to the NETL baseline cases. The reference plants were used in combination with proprietary equipment estimates to build a plant capital cost and cost of electricity evaluation. FPO performed better than the sub-bituminous post-combustion capture cases. Further preliminary estimates of improvements to the FPO cycle show even further gains when compared to conventional carbon capture methods.

20 FOSSIL-FUELED POWER PLANTS↗
DOE OSTI2025

Intelligent experiments through real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and future EIC detectors

This R&D project, initiated by the DOE Nuclear Physics AI-Machine Learning initiative in 2022, leverages AI to address data processing challenges in high-energy nuclear experiments (RHIC, LHC, and future EIC). Our focus is on developing a demonstrator for real-time processing of high-rate data streams from sPHENIX experiment tracking detectors. The limitations of a 15 kHz maximum trigger rate imposed by the calorimeters can be negated by intelligent use of streaming technology in the tracking system. The approach efficiently identifies low momentum rare heavy flavor events in high-rate p+p collisions (3MHz), using Graph Neural Network (GNN) and High Level Synthesis for Machine Learning (hls4ml). Success at sPHENIX promises immediate benefits, minimizing resources and accelerating the heavy-flavor measurements. The approach is transferable to other fields. For the EIC, we develop a DIS-electron tagger using Artificial Intelligence - Machine Learning (AI-ML) algorithms for real-time identification, showcasing the transformative potential of AI and FPGA technologies in high-energy nuclear and particle experiments real-time data processing pipelines.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
NASA NTRS2006

Low-Complexity Adaptive Lossless Compression of Hyperspectral Imagery

A low-complexity, adaptive predictive technique for lossless compression of hyperspectral imagery is described. This technique is designed to be suitable for implementation in hardware such as a field programmable gate array (FPGA); such an implementation could be used for high-speed compression of hyperspectral imagery onboard a spacecraft. The predictive step of the technique makes use of the sign algorithm, which is a relative of the least mean square (LMS) algorithm from the field of low-complexity adaptive filtering. The compressed data stream consists of prediction residuals encoded using a method similar to that of the JPEG-LS lossless image compression standard. Compression results are presented for several datasets including some raw Airborne Visible/ Infrared Imaging Spectrometer (AVIRIS) datasets and raw Atmospheric Infrared Sounder (AIRS) datasets. The compression effectiveness obtained with the technique is competitive with that of the best of previously described techniques with similar complexity.

lossless data compression↗
DOE OSTI2022

Probing Electron Beam Induced Transformations on a Single-Defect Level via Automated Scanning Transmission Electron Microscopy

A robust approach for real-time analysis of the scanning transmission electron microscopy (STEM) data streams, based on ensemble learning and iterative training (ELIT) of deep convolutional neural networks, is implemented on an operational microscope, enabling the exploration of the dynamics of specific atomic configurations under electron beam irradiation via an automated experiment in STEM. Combined with beam control, this approach allows studying beam effects on selected atomic groups and chemical bonds in a fully automated mode. Here, in this study, we demonstrate atomically precise engineering of single vacancy lines in transition metal dichalcogenides and the creation and identification of topological defects in graphene. The ELIT-based approach facilitates direct on-the-fly analysis of the STEM data and engenders real-time feedback schemes for probing electron beam chemistry, atomic manipulation, and atom by atom assembly.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
NASA NTRS2012

The HYSPIRI Decadal Survey Mission: Update on the Mission Concept and Science Objectives for Global Imaging Spectroscopy and Multi-Spectral Thermal Measurements

The NASA HyspIRI mission is planned to provide global solar reflected energy spectroscopic measurement of the terrestrial and shallow water regions of the Earth every 19 days will all measurements downlinked. In addition, HyspIRI will provide multi-spectral thermal measurements with a single band in the 4 micron region and seven bands in the 8 to 12 micron region with 5 day day/night coverage. A direct broadcast capability for measurement subsets is also planned. This HyspIRI mission is one of those designated in the 2007 National Research Council (NRC) Decadal Survey: Earth Science and Applications from Space. In the Decadal Survey, HyspIRI was recognized as relevant to a range of Earth science and science applications, including climate: "A hyperspectral sensor (e.g., FLORA) combined with a multispectral thermal sensor (e.g., SAVII) in low Earth orbit (LEO) is part of an integrated mission concept [described in Parts I and II] that is relevant to several panels, especially the climate variability panel." The HyspIRI science study group was formed in 2008 to evaluate and refine the mission concept. This group has developed a series of HyspIRI science objectives: (1) Climate: Ecosystem biochemistry, condition & feedback; spectral albedo; carbon/dust on snow/ice; biomass burning; evapotranspiration (2) Ecosystems: Global plant functional types, physiological condition, and biochemistry including agricultural lands (3) Fires: Fuel status, fire frequency, severity, emissions, and patterns of recovery globally (4) Coral reef and coastal habitats: Global composition and status (5) Volcanoes: Eruptions, emissions, regional and global impact (6) Geology and resources: Global distributions of surface mineral resources and improved understanding of geology and related hazards These objectives are achieved with the following measurement capabilities. The HyspIRI imaging spectrometer provides: full spectral coverage from 380 to 2500 at 10 nm sampling; 60 m spatial sampling with a 150 km swath; and fully downlinked coverage of the Earth's terrestrial and shallow water regions every 19 days to provide seasonal cloud-free coverage of the terrestrial surface. The HyspIRI Multi-Spectral Thermal instrument provides: 8 spectral bands from 4 to 12 microns; 60 m spatial sampling with a 600 km swath; and fully downlinked coverage of the Earth's terrestrial shallow water regions every 5 days (day/night) to provide nominally cloud-free monthly coverage. The HyspIRI mission also includes an on-board processing and direct broadcast capability, referred to as the Intelligent Payload Module (IPM), which will allow users with the appropriate antenna to download a subset of the HyspIRI data stream to a local ground station. These science and science application objectives are critical today and uniquely addressed by the combined imaging spectroscopy, thermal infrared measurements, and IPM direct broadcast capability of HyspIRI. Two key objectives are: (1) The global HyspIRI spectroscopic measurements of the terrestrial biosphere including vegetation composition and function to constrain and reduce the uncertainty in climate-carbon interactions and terrestrial biosphere feedback. (2) The global 8 band thermal measurements to provide improved constraint of fire related emissions. In this paper the current HyspIRI mission concept that has been reviewed and refined to its current level of maturity with a Data Products Symposium, Science Workshop and NASA HWorkshop is presented including traceability between the measurements and the science and science application objectives.

Green, Robert O.↗
DOE OSTI2026

Heterogeneous Multi-Domain Dataset Synthesis to Facilitate Privacy and Risk Assessments in Smart City IoT

The emergence of the Smart Cities paradigm and the rapid expansion and integration of Internet of Things (IoT) technologies within this context have created unprecedented opportunities for high-resolution behavioral analytics, urban optimization, and context-aware services. However, this same proliferation intensifies privacy risks, particularly those arising from cross-modal data linkage across heterogeneous sensing platforms. To address these challenges, this paper introduces a comprehensive, statistically grounded framework for generating synthetic, multimodal IoT datasets tailored to Smart City research. The framework produces behaviorally plausible synthetic data suitable for preliminary privacy risk assessment and as a benchmark for future re-identification studies, as well as for evaluating algorithms in mobility modeling, urban informatics, and privacy-enhancing technologies. As part of our approach, we formalize probabilistic methods for synthesizing three heterogeneous and operationally relevant data streams—cellular mobility traces, payment terminal transaction logs, and Smart Retail nutrition records—capturing the behaviors of a large number of synthetically generated urban residents over a 12-week period. The framework integrates spatially explicit merchant selection using K-Dimensional (KD)-tree nearest-neighbor algorithms, temporally correlated anchor-based mobility simulation reflective of daily urban rhythms, and dietary-constraint filtering to preserve ecological validity in consumption patterns. In total, the system generates approximately 116 million mobility pings, 5.4 million transactions, and 1.9 million itemized purchases, yielding a reproducible benchmark for evaluating multimodal analytics, privacy-preserving computation, and secure IoT data-sharing protocols. To show the validity of this dataset, the underlying distributions of these residents were successfully validated against reported distributions in published research. We present preliminary uniqueness and cross-modal linkage indicators; comprehensive re-identification benchmarking against specific attack algorithms is planned as future work. This framework can be easily adapted to various scenarios of interest in Smart Cities and other IoT applications. By aligning methodological rigor with the operational needs of Smart City ecosystems, this work fills critical gaps in synthetic data generation for privacy-sensitive domains, including intelligent transportation systems, urban health informatics, and next-generation digital commerce infrastructures.

IoT↗
NASA NTRS2011

Prioritized LT Codes

The original Luby Transform (LT) coding scheme is extended to account for data transmissions where some information symbols in a message block are more important than others. Prioritized LT codes provide unequal error protection (UEP) of data on an erasure channel by modifying the original LT encoder. The prioritized algorithm improves high-priority data protection without penalizing low-priority data recovery. Moreover, low-latency decoding is also obtained for high-priority data due to fast encoding. Prioritized LT codes only require a slight change in the original encoding algorithm, and no changes at all at the decoder. Hence, with a small complexity increase in the LT encoder, an improved UEP and low-decoding latency performance for high-priority data can be achieved. LT encoding partitions a data stream into fixed-sized message blocks each with a constant number of information symbols. To generate a code symbol from the information symbols in a message, the Robust-Soliton probability distribution is first applied in order to determine the number of information symbols to be used to compute the code symbol. Then, the specific information symbols are chosen uniform randomly from the message block. Finally, the selected information symbols are XORed to form the code symbol. The Prioritized LT code construction includes an additional restriction that code symbols formed by a relatively small number of XORed information symbols select some of these information symbols from the pool of high-priority data. Once high-priority data are fully covered, encoding continues with the conventional LT approach where code symbols are generated by selecting information symbols from the entire message block including all different priorities. Therefore, if code symbols derived from high-priority data experience an unusual high number of erasures, Prioritized LT codes can still reliably recover both high- and low-priority data. This hybrid approach decides not only "how to encode" but also "what to encode" to achieve UEP. Another advantage of the priority encoding process is that the majority of high-priority data can be decoded sooner since only a small number of code symbols are required to reconstruct high-priority data. This approach increases the likelihood that high-priority data is decoded first over low-priority data. The Prioritized LT code scheme achieves an improvement in high-priority data decoding performance as well as overall information recovery without penalizing the decoding of low-priority data, assuming high-priority data is no more than half of a message block. The cost is in the additional complexity required in the encoder. If extra computation resource is available at the transmitter, image, voice, and video transmission quality in terrestrial and space communications can benefit from accurate use of redundancy in protecting data with varying priorities.

Woo, Simon S.↗
DOE OSTI2024

Automated Framework for Groundwater Monitoring Using DWT with LSTM and Transformers

Environmental monitoring is critical for safeguarding public health and ecological well-being. Traditional data structuring and workflow monitoring methods consume significant time and effort, hindering timely insights and effective decision-making. Our study addresses this challenge by presenting an AI framework that automates data cleaning, structuring, and modeling processes, specifically targeting applications in groundwater monitoring. By leveraging automation for data processing and model training, our framework establishes a novel and efficient paradigm for environmental monitoring, with its potential application to the vast network of over a hundred Department of Energy Environmental Management (DoE-EM) cleanup sites across the country. It analyzes data streams from a network of groundwater Internet-of-Things (IoT) sensors deployed at the Savannah River Site (SRS) for prediction modeling. This allows human experts to focus on analysis and decision-making, ultimately leading to better environmental outcomes.The framework employs multivariate time-series forecasting methods to study and model the behavior of varying chemical analytes. The continuous learning process is enabled by utilizing deep learning techniques. It allows the framework to become more nuanced in its analysis over time, adapting to the specific characteristics of the environmental site and the evolving nature of contaminant behavior. Deep learning models known for sequence modeling, LSTM, and Transformers are employed for time series forecasting. Data processing and structuring are essential components significantly impacting the final model's performance. This hypothesis was proven by presenting a comparative analysis of model performance with processed and unprocessed data. The feature engineering approach utilized was the Discrete Wavelet Transform, which works well with time series data.

Discrete Wavelet Transform (DWT)↗