Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming data”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

The Phase-2 Upgrade of the CMS Data Acquisition

The High Luminosity LHC (HL-LHC) will start operating in 2027 after the third Long Shutdown (LS3), and is designed to provide an ultimate instantaneous luminosity of 7:5 × 10$^{34}$ cm$^{-2}$ s$^{-1}$, at the price of extreme pileup of up to 200 interactions per crossing. The number of overlapping interactions in HL-LHC collisions, their density, and the resulting intense radiation environment, warrant an almost complete upgrade of the CMS detector. The upgraded CMS detector will be read out by approximately fifty thousand highspeed front-end optical links at an unprecedented data rate of up to 80 Tb/s, for an average expected total event size of approximately 8 - 10 MB. Following the present established design, the CMS trigger and data acquisition system will continue to feature two trigger levels, with only one synchronous hardware-based Level-1 Trigger (L1), consisting of custom electronic boards and operating on dedicated data streams, and a second level, the High Level Trigger (HLT), using software algorithms running asynchronously on standard processors and making use of the full detector data to select events for offline storage and analysis. The upgraded CMS data acquisition system will collect data fragments for Level-1 accepted events from the detector back-end modules at a rate up to 750 kHz, aggregate fragments corresponding to individual Level- 1 accepts into events, and distribute them to the HLT processors where they will be filtered further. Events accepted by the HLT will be stored permanently at a rate of up to 7.5 kHz. This paper describes the baseline design of the DAQ and HLT systems for the Phase-2 of CMS.

Badaro, Gilbert↗
DOE OSTI2024

Towards a study protocol: A data-driven workflow to identify error sources in direct ink write mechatronics

Abstract Using Direct Ink Write (DIW) technology in a rapid and large-scale production requires reliable quality control for printed parts. Data streams generated during printing, such as print mechatronics, are massive and diverse which impedes extracting insights. In our study protocol approach, we developed a data-driven workflow to understand the behavior of sensor-measured X- and Y- axes positional errors with process parameters, such as print velocity and velocity control. We uncovered patterns showing that instantaneous changes in the velocity, when the build platform accelerates and decelerates, largely influence the positional errors, especially in the X- axis due to the hardware architecture. Since DIW systems share similar mechatronic inputs and outputs, our study protocol approach is broadly applicable and scalable across multiple systems. Graphical abstract

36 MATERIALS SCIENCE↗
DOE OSTI2022

Apparatus and amendment of wind turbine blade impact detection and analysis

A multisensory system provides both temporal and spatial coverage capacities for auto-detection of bird collision events. The system includes an apparatus having a first circuitry to capture and store a series of images or video of a blade of a wind turbine; and a memory to store the images from the first circuitry. The apparatus also has one or more sensors to continuously sense vibration of the blade or for acoustic recordings; and a second circuitry to analyze the sensor data stream and/or the series of images or video to identify a cause of the vibration and to trigger the camera(s). A communication interface transmits data from the second circuitry to another device, wherein the second circuitry applies artificial intelligence or machine learning to control sensitivity of the one or more sensors.

Johnston, Matthew↗
DOE OSTI2025

Real-time Anomaly Detection for Liquid Argon Time Projection Chambers

We present a real-time anomaly detection framework for liquid argon time projection chambers (LArTPCs), targeting applications in particle physics experiments such as the Short Baseline Near Detector (SBND) or the future Deep Underground Neutrino Experiment (DUNE). These experiments employ detectors that generate and stream high-resolution but sparse images of neutrino and other particle interactions. Our approach utilizes anomaly detection with autoencoders, compressed through knowledge distillation (KD), to enable the detection of anomalous signals in the data through efficient inference on resource-constrained hardware. The framework is targeted for deployment on computing platforms equipped with field-programmable gate arrays (FPGAs), GPUs, or CPUs, allowing low-latency selection of relevant activity directly from the raw detector data stream. We demonstrate that our approach is suitable for the detection and localization of anomalously "high-multiplicity" activity, and outline promising applications for LArTPC online data filtering and triggering.

FOS: Physical sciences↗
DOE OSTI2023

Machine Learning Enabled Sensor Fusion for In-Situ Defect Detection in Laser Powder Bed Fusion

Laser Powder Bed Fusion (L-PBF) Additive Manufacturing (AM) is among the metal 3D printing technologies most broadly adopted by the manufacturing industry. The current industry qualification paradigm for critical-application L-PBF parts relies heavily on expensive non-destructive inspection techniques such as X-Ray Computed Tomography (XCT), which significantly limits the use-cases of L-PBF. In situ monitoring of the process promises a less expensive alternative to ex situ testing, but existing sensor technologies and data analysis techniques struggle to detect sub-surface flaws (e.g., porosity and cracking) on production-scale L-PBF printers. RTX Technologies Research Center (RTRC) has licensed ORNL’s Peregrine software package – a printer- and camera-agnostic data analytics tool designed specifically for detecting process anomalies using in situ data collected during powder bed printing. The goal of this project was to feed temporally rich, multi-modal sensor data, including visible light, integrated near infrared (NIR), and spatially mapped co-axial melt pool thermal emission data into Peregrine to enable detection of subsurface flaws. XCT data was used as ground truth training data to allow Peregrine’s deep learning algorithms to recognize anomalies in these complex data streams in both test artifacts and industrially relevant geometries. Completion of this program has seen the successful implementation of multi-modal, multi-layer sensor data footprints for training of machine learning models in Peregrine. Flaws detected in XCT data have been successfully detected directly from this in situ data footprint, and initial analyses of the in situ probability-of-detection has been conducted, showing performance levels commensurate with traditional non-destructive evaluation (NDE) methods. The in situ monitoring methodology was then applied to an industrially relevant component that was using post-build NDE, highlighting the utility of the proposed method for hard-to-inspect AM components. As a direct result of this program, two journal manuscripts [1], [2] have been published in Additive Manufacturing, with additional manuscripts planned following program completion.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Measurement of charged-current $\nu_{\mu}$ and $\nu_e$ interactions with wire-cell in MicroBooNE towards a search for low-energy $\nu_e$ excess

This technote summarizes the existing work in searching for $\nu_e$ low-energy excess (eLEE) in MicroBooNE Booster Neutrino Beam (BNB) data stream based on the Wire-Cell event reconstruction paradigm. The charged-current $\nu_{\mu}$ and $\nu_e$ events are selected from the 5.3e19 POT data from the BNB beam and 2.06e20 POT data from Neutrinos at the Main Injector (NuMI) beam. The charged-current $\nu_e$ selection results from the BNB data that are sensitive to the eLEE search are not included. Various comparisons between data and Monte Carlo predictions are performed to validate the overall model and demonstrate the power of the analysis techniques.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRS1979

Solar maximum ultraviolet spectrometer and polarimeter

The objectives of the UVSP experiment are to study solar ultraviolet radiations, particularly from flares and active regions, and to measure constituents in the terrestrial atmosphere by the extinction of sunlight at satellite dawn and dusk. The instrument is designed to observe the Sun at a variety of spectral and spatial resolutions in the range from 1150 to 3600 A. A Gregorian telescope with effective focal length of 1.8 m is used to feed a 1 m Ebert-Fastie spectrometer. A polarimeter containing rotatable magnesium fluoride waveplates is included behind the spectrometer entrance slit and will allow all four Stokes parameters to be determined. Velocities on the Sun can also be measured. The instrument is controlled by a computer which can interact with the data stream to modify the observing program. The observing modes, including rasters, spectral scans, velocity measurements, and polarimetry, are also described along with plans for mission operations, data handling, and analysis of the observations.

Tandberg-Hanssen, E.↗
NASA NTRS2011

Low Latency DESDynI Data Products for Disaster Response, Resource Management and Other Applications

We are developing onboard processor technology targeted at the L-band SAR instrument onboard the planned DESDynI mission to enable formation of SAR images onboard opening possibilities for near-real-time data products to augment full data streams. Several image processing and/or interpretation techniques are being explored as possible direct-broadcast products for use by agencies in need of low-latency data, responsible for disaster mitigation and assessment, resource management, agricultural development, shipping, etc. Data collected through UAVSAR (L-band) serves as surrogate to the future DESDynI instrument. We have explored surface water extent as a tool for flooding response, and disturbance images on polarimetric backscatter of repeat pass imagery potentially useful for structural collapse (earthquake), mud/land/debris-slides etc. We have also explored building vegetation and snow/ice classifiers, via support vector machines utilizing quad-pol backscatter, cross-pol phase, and a number of derivatives (radar vegetation index, dielectric estimates, etc.). We share our qualitative and quantitative results thus far.

applications↗
NASA NTRSDate not supplied

Generation of Land Surface Reflectance with Combined Geo-KOMPSAT-2A AMI and Himawari 8 AHI Observations

The latest generation of geostationary satellites has opened a new era of Earth observations with unprecedented spatiotemporal resolution and spectral range. Together with GOES 16/17 ABI, FY4-A AGRI, and Himawari-8 AHI, a new Korean geostationary satellite (Geo-KOMPSAT-2A AMI) has operationally collected a full-disk image in 16 channels every ten minutes since July 2019, allowing diurnal land surface monitoring over a large proportion of Asia and all of Oceania. Retrieving accurate surface reflectance (SR) over land from GK-2A/AMI is a challenging but high priority objective. One of the challenges is the absence of a spectral band in the 2.2 m SWIR range from AMI, which is required by many atmospheric correction algorithms to retrieve atmospheric aerosol properties. To remedy this issue, we adopt a strategy that combines concurrent GK-2A/AMI and Himawari 8/AHI observations in order to derive AMI SR. We have adapted the NASA Multi-Angle Implementation of Atmospheric Correction (MAIAC) algorithm to process the data stream from Himawari 8/AHI. The advantages of the MAIAC algorithm is its capability to exploit the high temporal frequency and varying illumination geometry of the geostationary data for advanced cloud/snow detection, aerosol retrieval, and characterization of surface bidirectional reflectance properties. Leveraging the similarities of spectral bands and the sun-target-sensor geometry between AMI and AHI, we are able to create denser time series of observations and enhanced BRDF samples over most of the spatial coverage of AMI (and AHI). The combined stereo-type observations not only help derive SR for AMI but also enhance retrievals of the corresponding AHI surface products. We evaluate the resulting AMI SR using ground (AERONET) observations and corresponding MODIS products. Further, we discuss potential challenges in utilizing the geostationary satellite data for land surface monitoring.

geostationary satellite↗
NASA NTRSDate not supplied

Actor-based Runtime Verification with MESA

This work presents a runtime verification approach implemented in the tool MESA (MEssage-based System Analysis) which allows for using concurrent monitors to check for properties specified in data parameterized temporal logic and state machines. The tool is implemented as an internal Scala DSL. We employ the actor programming model to implement MESA where monitors are captured by concurrent actors that communicate via messaging. The paper presents a case study in which MESA is used to effectively monitor a large number of flights from live US airspace data streams. We also perform an empirical study by conducting experiments using monitoring systems with different numbers of concurrent monitors and different layers of indexing on the data contained in events. The paper describes the experiments, evaluates the results, and discusses challenges faced during the study. The evaluation shows the value of combining concurrency with indexing to handle data rich events.

runtime verification↗
DOE OSTI2023

ESnet/JLab FPGA Accelerated Transport

To increase the science rate for high data rates/volumes, Thomas Jefferson National Accelerator Facility (JLab) has partnered with Energy Sciences Network (ESnet) to define an edge to data center traffic shaping / steering transport capability featuring data event aware network shaping and forwarding. The keystone of this ESnet+JLab FPGA Accelerated Transport (EJFAT) is the joint development of an AI/ML directed dynamic compute work Load Balancer (LB) of UDP streamed data. The LB is a suite consisting of a Field Programmable Gate Array (FPGA) executing the dynamically configurable, low fixed latency LB data plane featuring real-time packet redirection and high throughput, and a control plane running on the FPGA host computer that monitors network and compute farm telemetry in order to make dynamic AI/ML guided decisions for destination compute host redirection/load balancing and destination resource provisioning. The LB provides for three-tier horizontal scaling across LB suites, core compute hosts, and CPUs within a host. The LB effectively provides seamless integration of edge/core computing to support direct experimental data processing for immediate use by JLab science programs and others such as the EIC as well as data centers of the future requiring high throughput and low latency for both hot and cooled data for both running experiment data acquisition systems and data center use cases.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Measurement of Charged-Current $ν_μ$ and $ν_e$ Interactions Towards a Search for Low-Energy $ν_Ε$ Excess Using Wire-Cell In Microboone

This technote summarizes the existing work in searching for $ν_e$ low-energy excess (eLEE) in MicroBooNE Booster Neutrino Beam (BNB) data stream based on the Wire-Cell event reconstruction paradigm. The charged-current $ν_μ$ and $ν_e$ events are selected from the 5.3e19 POT open data from the BNB beam, 6.37e20 POT far sideband from the BNB beam data, and 2.10e20 POT data from Neutrinos at the Main Injector (NuMI) beam. The charged-current $ν_e$ selection results from the BNB data that are sensitive to the eLEE search are not included. Various comparisons between data and Monte Carlo predictions are performed to validate the overall model and demonstrate the power of the analysis techniques. Physics sensitivities in terms of the exclusion and the discovery potential are presented.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
NASA NTRS1984

Revised radiometric calibration technique for LANDSAT-4 Thematic Mapper data by the Canada Centre for Remote Sensing

Observations of raw image data, raw radiometric calibration data, and background measurements extracted from the raw data streams on high density tape reveal major shortcomings in a technique proposed by the Canadian Center for Remote Sensing in 1982 for the radiometric correction of TM data. Results are presented which correlate measurements of the DC background with variations in both image data background and calibration samples. The effect on both raw data and data corrected using the earlier proposed technique is explained and the correction required for these factors as a function of individual scan line number for each detector is described. How the revised technique can be incorporated into an operational environment is demonstrated.

Murphy, J.↗
NASA NTRS2011

Validation of Real-Time Data Processing for the Ground and Air-MSPI Systems

JPL is currently developing the multi-angle spectro-polarimetric imager (MSPI), targeted for the Aerosol-Cloud-Ecosystems (ACE) mission, as defined in the National Academies 2007 Decadal Survey. In preparation for the space instrument, the MSPI team has built two incremental camera systems (Ground- and Air-MSPI) to improve understanding of the proposed architecture. Ground-MSPI is a gimballed instrument used primarily for stationary observation and characterization of the imager and optics. The ER-2 based Air-MSPI operates in a step-and-stare mode, providing multi-angle imaging of a static target. This mode-of-operation simulates the observation scenario of the space instrument. Physically, MSPI is a pushbroom camera with a specialized frontend. Before imaging, light entering the camera passes through a pair of photoelastic modulators and a set of pattern polarizers. These optical elements act on the light to make polarimetric extraction computationally feasible. Calculating polarimetric parameters from the imager's data stream requires a real-time least-squares computation that produces coefficients of a truncated time-series expansion of the image. As reported in, the data processing algorithm can operate in real-time on a Xilinx Virtex-5 FPGA. Moving beyond verification with an onboard data source, the algorithm has been validated on a commercial development board interfaced with the ground camera. In addition, the algorithm has been instantiated within the Air-MSPI electronics board's FPGA, and in situ first-light has been achieved.

real-time data processing↗
NASA NTRS2020

Actor-based Runtime Verification with MESA

This work presents a runtime verification approach implemented in the tool MESA (MEssage-based System Analysis) which allows for using concurrent monitors to check for properties specified in data parameterized temporal logic and state machines. The tool is implemented as an internal Scala DSL. We employ the actor programming model to implement MESA where monitors are captured by concurrent actors that communicate via messaging. The paper presents a case study in which MESA is used to effectively monitor a large number of flights from live US airspace data streams. We also perform an empirical study by conducting experiments using monitoring systems with different num- bers of concurrent monitors and different layers of indexing on the data contained in events. The paper describes the experiments, evaluates the results, and discusses challenges faced during the study. The evaluation shows the value of combining concurrency with indexing to handle data rich events.

Mehlitz, Peter↗
NASA NTRS2017

Framework for Processing Citizens Science Data for Applications to NASA Earth Science Missions

Citizen science (or crowdsourcing) has drawn much high-level recent and ongoing interest and support. It is poised to be applied, beyond the by-now fairly familiar use of, e.g., Twitter for natural hazards monitoring, to science research, such as augmenting the validation of NASA earth science mission data. This interest and support is seen in the 2014 National Plan for Civil Earth Observations, the 2015 White House forum on citizen science and crowdsourcing, the ongoing Senate Bill 2013 (Crowdsourcing and Citizen Science Act of 2015), the recent (August 2016) Open Geospatial Consortium (OGC) call for public participation in its newly-established Citizen Science Domain Working Group, and NASA's initiation of a new Citizen Science for Earth Systems Program (along with its first citizen science-focused solicitation for proposals). Over the past several years, we have been exploring the feasibility of extracting from the Twitter data stream useful information for application to NASA precipitation research, with both "passive" and "active" participation by the twitterers. The Twitter database, which recently passed its tenth anniversary, is potentially a rich source of real-time and historical global information for science applications. The time-varying set of "precipitation" tweets can be thought of as an organic network of rain gauges, potentially providing a widespread view of precipitation occurrence. The validation of satellite precipitation estimates is challenging, because many regions lack data or access to data, especially outside of the U.S. and in remote and developing areas. Mining the Twitter stream could augment these validation programs and, potentially, help tune existing algorithms. Our ongoing work, though exploratory, has resulted in key components for processing and managing tweets, including the capabilities to filter the Twitter stream in real time, to extract location information, to filter for exact phrases, and to plot tweet distributions. The key step is to process the "precipitation" tweets to be compatible with satellite-retrieved precipitation data. These key components for processing and managing "precipitation" tweets (and additional ones to be developed) are not limited to precipitation, nor are they limited to the Twitter social medium. Indeed, to maximize the value of our work for NASA earth science programs, these components should be generalized and be part of an overall framework for processing citizen science data for science research. In this paper, we outline such a framework.

earth science satellite data↗
NASA NTRS1974

The engineering design integration (EDIN) system

A digital computer program complex for the evaluation of aerospace vehicle preliminary designs is described. The system consists of a Univac 1100 series computer and peripherals using the Exec 8 operating system, a set of demand access terminals of the alphanumeric and graphics types, and a library of independent computer programs. Modification of the partial run streams, data base maintenance and construction, and control of program sequencing are provided by a data manipulation program called the DLG processor. The executive control of library program execution is performed by the Univac Exec 8 operating system through a user established run stream. A combination of demand and batch operations is employed in the evaluation of preliminary designs. Applications accomplished with the EDIN system are described.

Glatt, C. R.↗
DOE OSTI2020

Scalable Data-Intensive Geocomputation: A Design for Real-Time Continental Flood Inundation Mapping

The convergence of data-intensive and extreme-scale computing enables an integrated software and data ecosystem for scientific discovery. Developments in this realm will fuel transformative research in data-driven interdisciplinary domains. Geocomputation provides computing paradigms in Geographic Information Systems (GIS) for interactive computing of geographic data, processes, models, and maps. Because GIS is data-driven, the computational scalability of a geocomputation workflow is directly related to the scale of the GIS data layers, their resolution and extent, as well as the velocity of the geo-located data streams to be processed. Unique in high user interactivity and low end-to-end latency requirements, geocomputation applications will dramatically benefit from the convergence of high-end data analytics (HDA) and high-performance computing (HPC). The application level challenge, however, is to identify and eliminate computational bottlenecks that arise along a geocomputation workflow. Indeed, poor scalability at any of the workflow components is detrimental to the entire end-to-end pipeline. Here, we study a large geocomputation use case in flood inundation mapping that handles multiple national-scale geospatial datasets and targets low end-to-end latency. We discuss benefits and challenges for harnessing both HDA and HPC for data-intensive geospatial data processing and intensive numerical modeling of geographic processes. We propose an HDA+HPC geocomputation architecture design that couples HDA (e.g., Spark)-based spatial data handling and HPC-based parallel data modeling. Key techniques for coupling HDA and HPC to bridge the two different software stacks are reviewed and discussed.

Liu, Yan↗