Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22
All sourcesNASA NTRSDOE OSTI
NASA NTRS1992

Commercial potential of remote sensing data from the Earth observing system

The purpose was to assess the market potential of remote sensing value-added products from the Earth Observing System (EOS) platform. Sensors on the EOS platform were evaluated to determine which qualities and capabilities could be useful to the commercial user. The approach was to investigate past and future satellite data distribution programs. A questionnaire was developed for use in a telephone survey. Based on the results of the survey of companies that add value to remotely sensed data, conversations with the principal investigators in charge of each EOS sensor, a study of past commercial satellite data ventures, and reading from the commercial remote sensing industry literature, three recommendations were developed: develop a strategic plan for commercialization of EOS data, define a procedure for commercial users within the EOS data stream, and develop an Earth Observations Commercial Applications Program-like demonstration program within NASA using EOS simulated data.

Merry, Carolyn J.↗
DOE OSTI2025

A digital twin platform for building performance monitoring and optimization: Performance simulation and case studies

Advancements in sensor technology, data analytics, affordable compute, and communication infrastructure have paved the way for Digital Twin technology in optimizing building operations and controls. This study presents the development of an open and interoperable web-based Digital Twin platform for integrating diverse data streams and facilitating effective user interactions. The platform utilizes modern technologies for the web framework and time-series data management, ensuring scalability and responsiveness. The backend supports seamless integration of diverse data sources and emulators, incorporating data from building sensors and meters, external weather Application Programming Interfaces, and advanced EnergyPlus simulation models of the building and its energy systems including the Distributed Energy Resources that are formulated in Functional Mockup Units. A simulation case study was conducted with FlexLab, a test facility on Lawrence Berkeley National Laboratory campus. The case study includes normal operations, Distributed Energy Resource integration, and power outage scenarios, to illustrate the Digital Twin’s ability to provide critical insights into energy performance and thermal resilience. The results demonstrated the platform’s potential as a decision-support tool for optimizing building energy performance and enhancing resilience against extreme weather events. Future work will focus on deploying the Digital Twin platform to a real building for field validation, extending its capabilities to cover more scenarios such as bidirectional Electric Vehicle interactions, and enhancing user engagement.

EnergyPlus↗
DOE OSTI2025

Pinch Heat Integration Tool (PIT) v0.9

The pinch heat integration tool (PIT) is a multi-module tool designed for identifying and analyzing thermal energy integration at facilities. It generates pinch and grand composite curves to identify optimal streams for heat exchange and industrial heat pumps. Users input heating and cooling stream data via an Excel sheet. The tool comprises four modules: 1. 'Main Pinch' for creating a pinch diagram using user-provided data. 2. 'GCC' for generating a grand composite curve to identify utility optimization opportunities. 3. 'Heat Exchanger' for analyzing stream combinations for heat exchange. 4. 'Heat Pump' for evaluating streams as sources and sinks for heat pumps and calculating the heat pump's coefficient of performance based on the selected refrigerant.

Karki, Unique [Lawrence Berkeley National Laborato↗
DOE OSTI2023

Workflows Community Summit 2022: A Roadmap Revolution

Scientific workflows have become integral tools in broad scientific computing use cases. Science discovery is increasingly dependent on workflows to orchestrate large and complex scientific experiments that range from the execution of a cloud-based data preprocessing pipeline to multi-facility instrument-to-edge-to-HPC computational workflows. Given the changing landscape of scientific computing (often referred to as a computing continuum) and the evolving needs of emerging scientific applications, it is paramount that the development of novel scientific workflows and system functionalities seek to increase the efficiency, resilience, and pervasiveness of existing systems and applications. Specifically, the proliferation of machine learning/artificial intelligence (ML/AI) workflows, need for processing large-scale datasets produced by instruments at the edge, intensification of near real-time data processing, support for long-term experiment campaigns, and emergence of quantum computing as an adjunct to HPC, have significantly changed the functional and operational requirements of workflow systems. Workflow systems now need to, for example, support data streams from the edge-to-cloud-to-HPC, enable the management of many small-sized files, allow data reduction while ensuring high accuracy, orchestrate distributed services (workflows, instruments, data movement, provenance, publication, etc.) across computing and user facilities, among others. Further, to accelerate science, it is also necessary that these systems implement specifications/standards and APIs for seamless (horizontal and vertical) integration between systems and applications, as well as enable the publication of workflows and their associated products according to the FAIR principles.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2003

Moving NASA Remote Sensing Data to the GIS Environment for Health Studies

There has been an increasing demand by the health community for improved data on many different environmental factors relevant to the links between the environment and disease occurrence and transmission. These data are important for GIS-based monitoring, risk mapping, and surveillance of epidemiological parameters on a large number of different spatial, temporal, and spectral resolutions. Accordingly, NASA is developing new approaches to data collection and distribution in order to improve access to multiple sources of data streams to increase spatial and temporal coverage. Methods are being developed to incorporate different, scalable capabilities to handle multiple data sources by adding, deleting and replacing components as required as well as associated tools for their management. An approach has been to search for innovative solutions focused on the creation, use and manipulation of data stored in many different archives. These include data transformation and combination as well as data and information tools that can assist the public health and science community to use existing and anticipated products in new and flexible ways. This presentation will provide an inventory of geophysical parameters derived from satellite remote sensing sensors that are useful for GIS-based public health studies. The presentation will also discuss the physical and scientific limitations of access to and use of these data for health applications such as resolution and format differences, lack of software interoperability, data access problems. Finally, there will be a summary of the recent steps the NASA program has taken to bring NASA-generated satellite products to a wider range of users in the GIS community.

Vicente, Gilberto A.↗
NASA NTRSDate not supplied

Improving Geolocation Accuracy of the Advanced Meteorological Imager on the GEO-KOMPSAT-2A

GeoNEX is a collaborative project led by scientists from NASA and many other international institutes to generate Earth monitoring products using data streams from the latest geostationary (GEO) sensors. Its consistent processing and common gridding systems can produce research-quality data products from GEO sensors and leverage GEO-GEO or GEO-LEO (low earth orbit) synergistic uses. Currently, GeoNEX has produced and disseminated L1G (geometrically corrected Level 1 products) from GOES 16/17 ABIs and Himawari-8 AHI, but a new Korean geostationary sensor (Advanced Meteorological Imager, AMI) onboard Geo-KOMPSAT-2A covering a large proportion of Asia and all of Oceania is in development. Our recent efforts on assessing geolocation accuracy in ABI and AHI suggest a nontrivial residual exists in both level 1B data with varying spatiotemporal patterns. The findings urge us to prioritize identifying and correcting geolocation residuals of AMI to generate accurate and consistent GeoNEX top-of-atmosphere (TOA) reflectance products and following processing chains. Here we implement a phase correlation correction approach to a visible band (0.64 μm, 500 m) using landmarks prepared from finer scale digital terrain models. We characterize spatiotemporal patterns (e.g., diurnal & daily) of geolocation residuals of AMI before and after correction. We then assess stability of datasets and quantify impact of unexpected geolocation errors on terrestrial monitoring. The geolocation corrected AMI data are further compared with GeoNEX AHI L1G products which are able to create unique stereo-type observations with AMI through leveraging the similarities of spectral bands and the sun-target-sensor geometry. Further, we discuss challenges in utilizing the GEO-GEO (e.g., AMI & AHI) satellite data for potential applications.

Geostationary Satellites↗
DOE OSTI2022

Mu2e DAQ and slow control systems

The Mu2e experiment at the Fermilab Muon Campus will search for the coherent neutrinoless conversion of a muon into an electron in the feld of an aluminum nucleus with a sensitivity improvement by a factor of 10,000 over existing limits. The Mu2e Trigger and Data Acquisition System (TDAQ) uses otsdaq as the online Data Acquisition System (DAQ) solution. Developed at Fermilab, otsdaq integrates both the artdaq DAQ and the art analysis frameworks for event transfer, fltering, and processing. otsdaq is an online DAQ software suite with a focus on fexibility and scalability and provides a multiuser, web-based, interface accessible through a web browser. The data stream from the detector subsystems is read by a software flter algorithm that selects events which are combined with the data fux coming from a Cosmic Ray Veto System. The Detector Control System (DCS) has been developed using the Experimental Physics and Industrial Control System (EPICS) open source platform for monitoring, controlling, alarming, and archiving. The DCS System has been integrated into otsdaq. A prototype of the TDAQ and the DCS systems has been built at Fermilab’s Feynman Computing Center. In this paper, we report on the progress of the integration of this prototype in the online otsdaq software.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2021

Data acquisition and slow control interface for the Mu2e experiment

The Mu2e experiment at the Fermilab Muon Campus will search for the coherent neutrinoless conversion of a muon into an electron in the field of an aluminum nucleus with a sensitivity improvement by a factor of 10000 over existing limits. The Mu2e Trigger and Data Acquisition System (TDAQ) uses otsdaq as the online Data Acquisition System (DAQ) solution. Developed at Fermilab, otsdaq integrates both the artdaq DAQ and the art analysis frameworks for event transfer, filtering, and processing. otsdaq is an online DAQ software suite with a focus on flexibility and scalability and provides a multi-user, web-based, interface accessible through a web browser. The data stream from the detector subsystems is read by a software filter algorithm that selects events which are combined with the data flux coming from a cosmic ray veto system. The Detector Control System (DCS) has been developed using the Experimental Physics and Industrial Control System (EPICS) open source platform for monitoring, controlling, alarming, and archiving. The DCS system has been integrated into otsdaq. A prototype of the TDAQ and the DCS systems has been built at Fermilab's Feynman Computing Center. In this study, we report on the progress of the integration of this prototype in the online otsdaq software.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2020

Evaluation of a streaming readout solution for Jefferson Lab experiments

Jefferson Lab (JLab) operates the CEBAF accelerator that provides a high energy (up to 12 GeV) and high current (up to 150 µA) polarized electron beam for fixed target experiments. JLab's mission is to provide substantial progress in understanding the Quantum Chromo Dynamics (QCD), the current theory of the spectrum and the dynamics of hadronic matter. Given the broad scientific program a general-purpose detector, CLAS12, has been designed and constructed. Such a detector requires a sophisticated trigger and current experiments use an on-line FPGA-based system that relies upon custom firmware and electronics both of which are difficult reconfigure from one experiment to the next. To overcome these challenges transitioning to a streaming readout has been proposed since it would allow a more flexible, easier to debug, software trigger to be developed. To facilitate this JLab is gaining experience with streaming readout systems and is actively working to design and test a streaming data acquisition architecture. Significant work has already been done to make existing front-end electronics compatible with operation in streaming mode. Software-wise, much work remains to be done but an existing framework, TriDAS, which was developed for underwater neutrino telescopes can, in principle, be used with the existing JLab hardware. A test stand is already in place in the INDRA DAQ development lab at JLab. This paper will present the development of this TriDAS based test system and it?s operation in-beam using parts of the CLAS12 detector to evaluate streaming readout for a large general purpose detector.

Battaglieri, Marco↗
NASA NTRS1976

Definition and fabrication of an airborne scatterometer radar signal processor

A hardware/software system which incorporates a microprocessor design and software for the calculation of normalized radar cross section in real time was developed. Interface is provided to decommutate the NASA ADAS data stream for aircraft parameters used in processing and to provide output in the form of strip chart and pcm compatible data recording.

Source record↗
NASA NTRS1985

User data management

The primary objective is to identify, develop, and demonstrate key data management technologies to support user access to Space Station data. To accomplish this objective, there are several technical challenges which must be addressed. The first is how to provide routine customer access to high volume, dynamic and distributed data bases. This access will encompass the functions of mission and payload planning and operations, data processing and analysis, and data archive and distribution. Secondly, there must be some analysis of architectures for handling high volume data streams like those expected from the Space Station. This analysis will examine the use of packetized versus non-packetized data formats, modular expansion capabilities, real-time versus non-real-time data processing, and the interfaces and architecture required to support telescience operations. The task will also determine benchmarks of performance capabilities for technology operations, such as varied data base structures, data access procedures, distributed data base design, and data base machines. Information is provided here in outline form.

Source record↗
NASA NTRSDate not supplied

Do Better Satellite Precipitation Algorithms Improve Landslide Hazard Assessment?

Satellites make it possible to estimate precipitation in near real time. Given the challenges of achieving global coverage by other means, these data are used widely. However, few systems for landslide hazard assessment rely on satellite precipitation estimates. This could be due in part to perceptions of accuracy, although latency, spatial resolution, and other factors may also be important. We test whether recent changes to data streams from the Global Precipitation Measurement mission (GPM) have improved its potential for use in landslide prediction. Specifically, we examine data produced by the Integrated Multi-satellitERetrievals for the GPM (IMERG) algorithm, which was upgraded to version 7 this year. IMERG relies upon other algorithms, including the Goddard Profiling Algorithm (GPROF) and the GPM Combined Radar-Radiometer Algorithm (CORRA). Many changes have been made during the switch from IMERG version 6 to version 7. These include upgrading CORRA and GPROF to version 7, to improve the accuracy of precipitation in frozen, mountainous, and coastal areas. The measured intensity of some storms has been enhanced with a new algorithm, the Scheme for Histogram Adjustment with Ranked Precipitation Estimates in the Neighborhood. Combined with many others, these changes to IMERG should improve its utility for landslide hazard assessment in a variety of contexts. To test this idea, we retrain the global Landslide Hazard Assessment for Situational Awareness (LHASA) model twice—first with data from IMERG version 6B and second with 7B. Since current daily rainfall is the most important variable in determining outcomes predicted by LHASA, it should reflect changes made to that input. First, we grid the landslides at a daily, thirty-arcsecond resolution. This serves as the response variable. At each of these sites current and antecedent rainfall are extracted, along with antecedent snow mass and soil moisture, slope, and PGA. In addition, one million grid cells are selected at random points to represent conditions under which landslides (probably) do not occur. After merging these data, we hold back 20% of the dataset for validation purposes and train a machine-learning model with the rest. We assess both the model’s overall ability to identify landslides and its ability to predict specific large landslide disasters.

Thomas A Stanley↗
DOE OSTI2025

Redis-Based Streaming Architecture for Accelerator Beam Instrumentation DAQ Systems

The Fermilab Acceleraor Division, Beam Instrumentation Department, is always adopting modern and current software methodologies for complex DAQ architectures. This paper highlights the Redis Adapter (RA) as the key software component enabling high performance, modular communication between digitizers and distributed control systems by leveraging Redis and containerization. The RA provides a unified, efficient interface between Redis based data streams and consumer systems. In the legacy architecture, digitized data flowed through the custom, UDP based Distributed Data Communication Protocol in the middle layer. In the current system, DDCP remains the ingestion path, while the RA serves as the decoupling layer. The proposed system replaces old VME digitizers with a SOM-based digitizer that communicates with Redis using the RA. The RA acts as both a performance-critical bridge and a protocol-agnostic adapter, ensuring compatibility with legacy control frameworks while enabling future scalability and modularity. This restructuring of the middle layer also helps the system achieve high throughput, reduce latency, and simplify the data path. Finally, we will demonstrate how RA is utilized in our two core products to deliver both legacy compatibility and future flexibility.

Joshi, S. [Fermilab]↗
DOE OSTI2022

Bridging microscopy with molecular dynamics and quantum simulations: an atomAI based pipeline

Recent advances in (scanning) transmission electron microscopy have enabled a routine generation of large volumes of high-veracity structural data on 2D and 3D materials, naturally offering the challenge of using these as starting inputs for atomistic simulations. In this fashion, the theory will address experimentally emerging structures, as opposed to the full range of theoretically possible atomic configurations. However, this challenge is highly nontrivial due to the extreme disparity between intrinsic timescales accessible to modern simulations and microscopy, as well as latencies of microscopy and simulations per se. Addressing this issue requires as a first step bridging the instrumental data flow and physics-based simulation environment, to enable the selection of regions of interest and exploring them using physical simulations. Here we report the development of the machine learning workflow that directly bridges the instrument data stream into Python-based molecular dynamics and density functional theory environments using pre-trained neural networks to convert imaging data to physical descriptors. Additionally, the pathways to ensure structural stability and compensate for the observational biases universally present in the data are identified in the workflow. This approach is used for a graphene system to reconstruct optimized geometry and simulate temperature-dependent dynamics including adsorption of Cr as an ad-atom and graphene healing effects. However, it is universal and can be used for other material systems.

36 MATERIALS SCIENCE↗
NASA NTRS1985

Extended testing of a general contextual classifier using the massively parallel processor - Preliminary results and test plans

Earlier encouraging test results of a contextual classifier that combines spatial and spectral information employing a general statistical approach are expanded. The earlier results were of limited meaning because they were produced from small (50-by-50 pixel) data sets. An implementation of the contextual classifier on NASA Goddard's Massively Parallel Processor (MPP) is presented; for the first time the MPP makes feasible the testing of the classifier on large data sets (a 12-hour test on a VAX-11/780 minicomputer now takes 5 minutes on the MPP). The MPP is a Single-Instruction, Multiple Data Stream computer, consisting of 16,384 bit serial microprocessors connected in a 128-by-128 mesh array with each element having data transfer connections with its four nearest neighbors so that the MPP is capable of billions of operations per second. Preliminary results are given (with more expected for the conference) and plans are mentioned for extended testing of the contextual classifier on Thematic Mapper data sets.

Tilton, J. C.↗
NASA NTRS2012

The Atacama Cosmology Telescope: Data Characterization and Map Making

We present a description of the data reduction and mapmaking pipeline used for the 2008 observing season of the Atacama Cosmology Telescope (ACT). The data presented here at 148 GHz represent 12% or the 90 TB collected by ACT from 2007 to 2010. In 2008 we observed for 136 days, producing a total of 142h of data (11 TB for the 148 GHz band only), with a daily average of 10.5 h of observation. From these, 108.5 h were devoted to 850 sq deg stripe (11.2 h by 9 deg.1) centered on a declination of -52 deg.7, while 175 h were devoted to a 280 square deg stripe (4.5 h by 4 deg.8) centered at the celestial equator. We discuss sources of statistical and systematic noise, calibration, telescope pointing and data selection. Out of 1260 survey hours and 1024 detectors per array, 816 h and 593 effective detectors remain after data selection for this frequency band, yielding a 38 % survey efficiency. The total sensitivity in 2008, determined from the noise level between 5 Hz and 20 Hz in the time-ordered data stream (TOD), is 32 muK square root of s in CMB units. Atmospheric brightness fluctuations constitute the main contaminant in the data and dominate the detector and noise covariance at low frequencies in the TOD. The maps were made by solving the lease squares problem using the Preconditioned Conjugate Gradient method, incorporating the details of the detector and noise correlations. Cross-correlation with WMAP sky maps as well as analysis from simulations reveal the our maps are unbiased at l > 300. This paper accompanies the public release of the 148 GHz southern stripe maps from 2008. The techniques described here will be applied to future maps and data releases.

Duenner, Rolando↗
NASA NTRS2013

Suomi NPP VIIRS Prelaunch and On-orbit Geometric Calibration and Characterization

The Visible Infrared Imager Radiometer Suite (VIIRS) sensor was launched 28 October 2011 on the Suomi National Polarorbiting Partnership (SNPP) satellite. VIIRS has 22 spectral bands covering the spectrum between 0.412 m and 12.01 m, including 16 moderate resolution bands (M-bands) with a spatial resolution of 750 m at nadir, 5 imaging resolution bands (I-bands) with a spatial resolution of 375 m at nadir, and 1 day-night band (DNB) with a near-constant 750 m spatial resolution throughout the scan. These bands are located in a visible and near infrared (VisNIR) focal plane assembly (FPA), a short- and mid-wave infrared (SWMWIR) FPA and a long-wave infrared (LWIR) FPA. All bands, except the DNB, are co-registered for proper environmental data records (EDRs) retrievals. Observations from VIIRS instrument provide long-term measurements of biogeophysical variables for climate research and polar satellite data stream for the operational communitys use in weather forecasting and disaster relief and other applications. Well Earth-located (geolocated) instrument data is important to retrieving accurate biogeophysical variables. This paper describes prelaunch pointing and alignment measurements, and the two sets of on-orbit correction of geolocation errors, the first of which corrected error from 1,300 m to within 75 m (20 I-band pixel size), and the second of which fine tuned scan angle dependent errors, bringing VIIRS geolocation products to high maturity in one and a half years of the SNPP VIIRS on-orbit operations. Prelaunch calibration and the on-orbit characterization of sensor spatial impulse responses and band-to-band co-registration (BBR) are also described.

Geolocation↗