Results for “streaming data” — Page 27

NASA NTRS1978

Data processing in infrared astronomy

Infrared astronomy is often carried out with rocket probes or orbiting satellite telescopes in order to escape the effects of atmospheric absorption. The data returned from such missions is a highly abstracted digital representation of measurements made by analog detectors. The ability to extract infrared-emission information from these data streams depends on a thorough understanding of the information flow from the telescope aperture to the computer center. This paper reviews the primary elements of this end-to-end concept and the impact of each of these elements on the data processing algorithms, including the division between onboard and ground processing for scientific measurements.

Pelzmann, R. F., Jr.↗

NASA NTRSDate not supplied

Datascope to Enable Earth Independent Medical Operations (EIMO)

BACKGROUND: NASA has amassed sixty years of knowledge and experience relevant to maintenance of crew health and performance in low earth orbit. The Apollo Program introduced the importance of ensuring progressively autonomous operational capability. Earth Independent Medical Operations (EIMO) will require a gradual shift in the balance of medical responsibility, management, and authority from terrestrial to space-based assets. Terrestrial assets will continue to be essential for pre-mission screening and planning in addition to maintenance of crew health and performance. However, new capabilities are needed to enable EIMO and the amount of data required to support these systems, and mitigate the impacts of data transmission delays and reduced bandwidth coupled with lack of cloud-like resources and on-board computing capacity that is currently unclear or operationally insufficient. OVERVIEW: The overall goal of EIMO is to develop artificial intelligence (AI)-based solutions to analyze crew health and performance data utilizing a clinical decision support system (CDSS) to provide crew medical officers (CMO) with the equivalent of real-time, on-board medical consults. The EIMO ecosystem is envisioned as a “system of systems” where embedded reference databases and real-time data streams from multiple input vectors continuously and seamlessly assess crew health and performance. EIMO will be designed to make recommendations to the CMO using multi-modal AI-based natural language processing and machine learning methods with interoperability to push/pull data within and between multiple vehicle and habitat architectures. DISCUSSION: Data flows and storage/retrieval capacity are severely constrained during space missions and the challenges will become even greater during exploration missions. Just as each past program from Mercury to the International Space Station (ISS) required rethinking the interaction between ground-based controllers and space-based crew, so too will future missions to the Moon and Mars. While the NASA High-Performance Spaceflight Computing Processor project aims to increase computational capacity by 100 times over current spaceflight computers, the projected deliverable still lags considerably behind what will be needed to enable an AI-driven CDSS. Restrictions in processing speed and data storage capacity, coupled with transmission bottlenecks and delays, necessitate definition and optimization of an integrated data architecture to enable a progressively autonomous medical capability.

Medical operations↗

DOE OSTI2021

Data Analysis Approach for Large Data Volumes in a Connected Community

Recent advancements within smart neighborhoods where utilities are enabling automatic control of appliances such as heating, ventilation, and air conditioning (HVAC) and water heater (WH) systems are providing new opportunities to minimize energy costs through reduced peak load. This requires systematic collection, storage, management, and in-memory processing of large volumes of streaming data for fast performance. In this paper, we propose a multi-tier layered IoT software framework that enables effective descriptive and predictive data analysis for understanding live operation of the neighborhood, fault identification, and future opportunities for further optimization of load curves. We then demonstrate how we achieve live situational awareness of the connected neighborhood through a suite of visualization components. Finally, we discuss a few analytic dashboards that address questions such as peak load reductions obtained due to optimization, customer preference for automatic control of appliances (do they override the automatic control of HVAC?, etc.). 1 1 This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Chinthavali, Supriya↗

NASA NTRS1989

Prototype architecture for a VLSI level zero processing system

The prototype architecture and implementation of a high-speed level zero processing (LZP) system are discussed. Due to the new processing algorithm and VLSI technology, the prototype LZP system features compact size, low cost, high processing throughput, and easy maintainability and increased reliability. Though extensive control functions have been done by hardware, the programmability of processing tasks makes it possible to adapt the system to different data formats and processing requirements. It is noted that the LZP system can handle up to 8 virtual channels and 24 sources with combined data volume of 15 Gbytes per orbit. For greater demands, multiple LZP systems can be configured in parallel, each called a processing channel and assigned a subset of virtual channels. The telemetry data stream will be steered into different processing channels in accordance with their virtual channel IDs. This super system can cope with a virtually unlimited number of virtual channels and sources. In the near future, it is expected that new disk farms with data rate exceeding 150 Mbps will be available from commercial vendors due to the advance in disk drive technology.

Shi, Jianfei↗

NASA NTRS2005

CDEP Consortium on Ocean Data Assimilation for Seasonal-to-Interannual Prediction (ODASI)

The ODASI consortium is focused activity of the NOAA/OGP/Climate Diagnostics and Experimental Prediction Program with the goal of improving ocean data assimilation methods and their implementations in support of seasonal forecasts with coupled general circulation models. The consortium is undertaking coordinated assimilation experiments, with common forcing data sets and common input data streams. With different assimilation systems and different models, we aim to understand what approach works best in improving forecast skill in the equatorial Pacific. The presentation will provide an overview of the consortium goals and plans and recent results focused towards evaluating data impacts.

Rienecker, Michele↗

NASA NTRS2020

Global Data Assembly Center (gdac) Report to the GHRSST Science Team

In 2019-2020 the Global Data Assembly Center (GDAC) at NASA’s Physical Oceanography Distributed Active Archive Center (PO.DAAC) provided ingest, archive, distribution and user services for GHRSST operational data streams with improved and evolved tools, services, and tutorials, and interfaced with the user community to address technical inquiries. Several new GHRSST datasets including reprocessed MODIS Aqua/Terra L2P with improved cloud screening were made available. The PO.DAAC participated in the evolving GHRSST data management re architecture and development activities. GHRSST Science Team member Edward Armstrong assumed the role of co-leader of the CEOS SST Virtual Constellation. The following sections summarize and document the specific achievements of the GDAC to the GHRSST community.

Finch, Chris↗

DOE OSTI2025

Enabling Seamless Transitions from Experimental to Production HPC for Interactive Workflows

The evolving landscape of scientific computing requires seamless transitions from experimental to production HPC environments for interactive workflows. This paper presents a structured transition pathway developed at OLCF that bridges the gap between development testbeds and production systems. We address both technological and policy challenges, introducing frameworks for data streaming architectures, secure service interfaces, and adaptive resource scheduling for time-sensitive workloads and improved HPC interactivity. Our approach transforms traditional batch-oriented HPC into a more dynamic ecosystem capable of supporting modern scientific workflows that require near real-time data analysis, experimental steering, and cross-facility integration.

Etz, Brian [ORNL] (ORCID:0000000208554863)↗

DOE OSTI2024

Atmospheric Radiation Measurement (ARM) airborne field campaign data products between 2013 and 2018

Airborne measurements are pivotal for providing detailed, spatiotemporally resolved information about atmospheric parameters and aerosol and cloud properties, thereby enhancing our understanding of dynamic atmospheric processes. For 30 years, the US Department of Energy (DOE) Office of Science supported an instrumented Gulfstream 1 (G-1) aircraft for atmospheric field campaigns. Data from the final decade of G-1 operations were archived by the Atmospheric Radiation Measurement (ARM) Data Center and made publicly available at no cost to all registered users. To ensure a consistent data format and to improve the accessibility of the ARM airborne data, an integrated dataset was recently developed covering the final 6 years of G-1 operations (2013 to 2018, https://doi.org/10.5439/1999133; Mei and Gaustad, 2024). The integrated dataset includes data collected from 236 flights (766.4 h), which covered the Arctic, the US Southern Great Plains (SGP), the US West Coast, the eastern North Atlantic (ENA), the Amazon Basin in Brazil, and the Sierras de Córdoba range in Argentina. These comprehensive data streams provide much-needed insight into spatiotemporal variability in the thermodynamic quantities and aerosol and cloud properties for addressing essential science questions in Earth system process studies. This paper describes the DOE ARM merged G-1 datasets, including information on the acquisition, data collection challenges and future potentials, and quality control processes. It further illustrates the usage of this merged dataset to evaluate the Energy Exascale Earth System Model (E3SM) with the Earth System Model Aerosol–Cloud Diagnostics (ESMAC Diags) package.

54 ENVIRONMENTAL SCIENCES↗

NASA NTRS2006

Wireless augmented reality communication system

The system of the present invention is a highly integrated radio communication system with a multimedia co-processor which allows true two-way multimedia (video, audio, data) access as well as real-time biomedical monitoring in a pager-sized portable access unit. The system is integrated in a network structure including one or more general purpose nodes for providing a wireless-to-wired interface. The network architecture allows video, audio and data (including biomedical data) streams to be connected directly to external users and devices. The portable access units may also be mated to various non-personal devices such as cameras or environmental sensors for providing a method for setting up wireless sensor nets from which reported data may be accessed through the portable access unit. The reported data may alternatively be automatically logged at a remote computer for access and viewing through a portable access unit, including the user's own.

Devereaux, Ann↗

NASA NTRS2012

NASA/SPoRT's GOES-R Activities in Support of Product Development, Management, and Training

The NASA Short-term Prediction Research and Transition (SPoRT) Center supports many activities within the GOES-R Proving Grounds (PG). These include the development of imagery from existing instrumentation as a proxy to future Advanced Baseline Imager (ABI) capabilities on GOES-R. The Moderate Resolution Imaging Spectroradiometer (MODIS) and the Visible/Infrared Imager/Radiometer Suite (VIIRS) instruments are used to provide a glimpse of the multi-spectral capabilities that will become the norm as the number of channels and data rate dramatically increase with GOES-R. The NOAA/NWS has plans to provide operational users with all ABI channels at the highest resolution. Data fusion of individual channels into composite red, green, and blue imagery products will assist the end user with this future wave of information. While increasing the efficiency in the operational use of ABI channels, these composites provide only qualitative information. Within the GOES-R PG, SPoRT and other partners are exploring ways to include quantitative information as part of the composite imagery. However, limitations in local hardware processing and/or data bandwidth for users of the GOES-R data stream are challenges to overcome. This presentation will discuss the creation of these composite images as well as possible solutions to address these processing challenges. In a similar manner the Geostationary Lightning Mapper (GLM) to be launched on GOES-R presents several data management challenges. The GLM is a pioneering instrument to quantify total lightning from a geostationary platform. The expected data frequency from the GLM is to be at a sub-minute interval. Users of such a data set may have little experience in handling such a rapid update of information. To assist users, SPoRT is working with the NWS to develop tools within the user fs decision support system to allow tracking and analysis of total lightning from a storm-based perspective. This presentation will discuss the challenges and progress of this tool development work. With new data and products comes the need for user Training. Within the GOES-R PG SPoRT is supporting the demonstration of these future products by providing various training materials to end users. A summary of training provided to operational users will be discussed.

Fuell, Kevin K.↗

NASA NTRS2014

Wireless Augmented Reality Communication System

The system of the present invention is a highly integrated radio communication system with a multimedia co-processor which allows true two-way multimedia (video, audio, data) access as well as real-time biomedical monitoring in a pager-sized portable access unit. The system is integrated in a network structure including one or more general purpose nodes for providing a wireless-to-wired interface. The network architecture allows video, audio and data (including biomedical data) streams to be connected directly to external users and devices. The portable access units may also be mated to various non-personal devices such as cameras or environmental sensors for providing a method for setting up wireless sensor nets from which reported data may be accessed through the portable access unit. The reported data may alternatively be automatically logged at a remote computer for access and viewing through a portable access unit, including the user's own.

Devereaux, Ann↗

NASA NTRS2014

Wireless Augmented Reality Communication System

The system of the present invention is a highly integrated radio communication system with a multimedia co-processor which allows true two-way multimedia (video, audio, data) access as well as real-time biomedical monitoring in a pager-sized portable access unit. The system is integrated in a network structure including one or more general purpose nodes for providing a wireless-to-wired interface. The network architecture allows video, audio and data (including biomedical data) streams to be connected directly to external users and devices. The portable access units may also be mated to various non-personal devices such as cameras or environmental sensors for providing a method for setting up wireless sensor nets from which reported data may be accessed through the portable access unit. The reported data may alternatively be automatically logged at a remote computer for access and viewing through a portable access unit, including the user's own.

Devereaux, Ann↗

NASA NTRS2016

Wireless Augmented Reality Communication System

The system of the present invention is a highly integrated radio communication system with a multimedia co-processor which allows true two-way multimedia (video, audio, data) access as well as real-time biomedical monitoring in a pager-sized portable access unit. The system is integrated in a network structure including one or more general purpose nodes for providing a wireless-to-wired interface. The network architecture allows video, audio and data (including biomedical data) streams to be connected directly to external users and devices. The portable access units may also be mated to various non-personal devices such as cameras or environmental sensors for providing a method for setting up wireless sensor nets from which reported data may be accessed through the portable access unit. The reported data may alternatively be automatically logged at a remote computer for access and viewing through a portable access unit, including the user's own.

Devereaux, Ann↗

NASA NTRS2017

Application of Machine Learning to Rotorcraft Health Monitoring

Machine learning is a powerful tool for data exploration and model building with large data sets. This project aimed to use machine learning techniques to explore the inherent structure of data from rotorcraft gear tests, relationships between features and damage states, and to build a system for predicting gear health for future rotorcraft transmission applications. Classical machine learning techniques are difficult, if not irresponsible to apply to time series data because many make the assumption of independence between samples. To overcome this, Hidden Markov Models were used to create a binary classifier for identifying scuffing transitions and Recurrent Neural Networks were used to leverage long distance relationships in predicting discrete damage states. When combined in a workflow, where the binary classifier acted as a filter for the fatigue monitor, the system was able to demonstrate accuracy in damage state prediction and scuffing identification. The time dependent nature of the data restricted data exploration to collecting and analyzing data from the model selection process. The limited amount of available data was unable to give useful information, and the division of training and testing sets tended to heavily influence the scores of the models across combinations of features and hyper-parameters. This work built a framework for tracking scuffing and fatigue on streaming data and demonstrates that machine learning has much to offer rotorcraft health monitoring by using Bayesian learning and deep learning methods to capture the time dependent nature of the data. Suggested future work is to implement the framework developed in this project using a larger variety of data sets to test the generalization capabilities of the models and allow for data exploration.

machine learning↗

DOE OSTI2023

Parallel Algorithms for Computing the Tensor-Train Decomposition

The tensor-train (TT) decomposition expresses a tensor in a data-sparse format used in molecular simulations, high-order correlation functions, and optimization. In this paper, we propose four parallelizable algorithms that compute the TT format from various tensor inputs: (1) Parallel-TTSVD for traditional format, (2) PSTT and its variants for streaming data, (3) Tucker2TT for Tucker format, and (4) TT-fADI for solutions of Sylvester tensor equations. We provide theoretical guarantees of accuracy, parallelization methods, scaling analysis, and numerical results. For example, for a d-dimension tensor in $\mathbb{R}$ $n\times∙∙∙$$\times$$n$ a two-sided sketching algorithm PSTT2 is shown to have a memory complexity of $O(n^{[d/2]})$, improving upon $O(n^{d—1})$ from previous algorithms.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1989

Fractal image compression

Fractals are geometric or data structures which do not simplify under magnification. Fractal Image Compression is a technique which associates a fractal to an image. On the one hand, the fractal can be described in terms of a few succinct rules, while on the other, the fractal contains much or all of the image information. Since the rules are described with less bits of data than the image, compression results. Data compression with fractals is an approach to reach high compression ratios for large data streams related to images. The high compression ratios are attained at a cost of large amounts of computation. Both lossless and lossy modes are supported by the technique. The technique is stable in that small errors in codes lead to small errors in image data. Applications to the NASA mission are discussed.

Barnsley, Michael F.↗

NASA NTRS2005

Support for the Harvard University Water Vapor and Total Water Instruments for the 2004 NASA WB57 Middle Latitude Cirrus Experiment

In order to improve our understanding of the role clouds play in the climate system, NASA is investing considerable effort in characterizing clouds with instruments ranging from passive remote sensors on board the EOS platforms, to the forthcoming active remote sensors on Cloudsat and Calipso. These missions, when taken together, have the capacity to advance our understanding of the coupling between various components of the hydrologic cycle and the atmospheric circulation, and hold the additional potential of leading to significant improvements in the characterization of cloud feedbacks in global models. This is especially true considering that several of these platforms will be flown in an identical orbit within several minutes of one another-a constellation of satellites known as the A-Train. The algorithms that are being implemented and developed to convert these new data streams from radiance and reflectivity measurements into geophysical parameters invariably rely on some set of simplifymg assumptions and empirical constants. Uncertainties in these relationships lead to poorly understood random and systematic errors in the retrieved properties. This lack of understanding introduces ambiguity in interpreting the data and in using the global data sets for their intended purposes. In light of this, a series of flights with the W57F was proposed to address certain specific issues related to the basic properties of mid latitude cirrus clouds: the NASA WE357 Middle Latitude Cirrus Experiment ("MidCiX"). The science questions addressed are: 1) Can cloud property retrieval algorithms developed for A-Train active and passive remote sensing measurements accurately characterize the microphysical properties of synoptic and convectively generated cirrus cloud systems? 2) What are the relationships between the cirrus particle mass, projected area, and particle size spectrum in various genre of cirrus clouds? 3) Does the present compliment of state of the art in situ cloud probes provide the level of precision and accuracy needed to develop and validate algorithms and to contribute to our understanding of the characteristics and microphysical processes operating in cirrus clouds?

Anderson, James G.↗

DOE OSTI2023

Improving an Acoustic Vehicle Detector Using an Iterative Self-Supervision Procedure

In many non-canonical data science scenarios, obtaining, detecting, attributing, and annotating enough high-quality training data is the primary barrier to developing highly effective models. Moreover, in many problems that are not sufficiently defined or constrained, manually developing a training dataset can often overlook interesting phenomena that should be included. To this end, we have developed and demonstrated an iterative self-supervised learning procedure, whereby models are successfully trained and applied to new data to extract new training examples that are added to the corpus of training data. Successive generations of classifiers are then trained on this augmented corpus. Using low-frequency acoustic data collected by a network of infrasound sensors deployed around the High Flux Isotope Reactor and Radiochemical Engineering Development Center at Oak Ridge National Laboratory, we test the viability of our proposed approach to develop a powerful classifier with the goal of identifying vehicles from continuously streamed data and differentiating these from other sources of noise such as tools, people, airplanes, and wind. Using a small collection of exhaustively manually labeled data, we test several implementation details of the procedure and demonstrate its success regardless of the fidelity of the initial model used to seed the iterative procedure. Finally, we demonstrate the method’s ability to update a model to accommodate changes in the data-generating distribution encountered during long-term persistent data collection.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗