Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data streams”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35

Bridging Cloud and Edge Computing at NREL Using CONNECT: Cloud Optimized Networking for Next-Gen Edge Computing Technologies [Slides]

CONNECT is an innovative on-premise hardware and software solution that integrates edge and cloud computing infrastructure at NREL. Built on the AWS Greengrass middleware and leveraging the MQTT protocol, CONNECT enables real-time data streaming from IoT devices and gateways to both cloud and local services, empowering researchers to rapidly capture, analyze, and act upon edge-generated data while leveraging cloud capabilities. The platform addresses research infrastructure challenges by providing a pre-approved platform which is already configured with the correct networking and cybersecurity baselines thus eliminating procurement delays and enabling on-demand availability. CONNECT's hybrid architecture efficiently manages burstable workloads, allowing research teams to dynamically scale computational capacity, handle peak data loads, and reduce operational bottlenecks. Advanced capabilities include built-in GPU support for executing machine learning models which enables low-latency inference at the edge from models trained in the cloud. This architecture supports real-time analytics and filtering, providing a mechanism to allow only transmitting and processing high-value data. Cloud-based configuration management permits engineers to manage on-premise systems remotely, optimizing operational efficiency. By bridging edge and cloud computing, CONNECT provides NREL researchers with a flexible, scalable platform that accelerates scientific discovery while maintaining robust security and performance standards.

97 MATHEMATICS AND COMPUTING↗

MIRAGE: The data acquisition, analysis, and display system

Developed for the NASA Johnson Space Center and Life Sciences Directorate by GE Government Services, the Microcomputer Integrated Real-time Acquisition Ground Equipment (MIRAGE) system is a portable ground support system for Spacelab life sciences experiments. The MIRAGE system can acquire digital or analog data. Digital data may be NRZ-formatted telemetry packets of packets from a network interface. Analog signal are digitized and stored in experimental packet format. Data packets from any acquisition source are archived to a disk as they are received. Meta-parameters are generated from the data packet parameters by applying mathematical and logical operators. Parameters are displayed in text and graphical form or output to analog devices. Experiment data packets may be retransmitted through the network interface. Data stream definition, experiment parameter format, parameter displays, and other variables are configured using spreadsheet database. A database can be developed to support virtually any data packet format. The user interface provides menu- and icon-driven program control. The MIRAGE system can be integrated with other workstations to perform a variety of functions. The generic capabilities, adaptability and ease of use make the MIRAGE a cost-effective solution to many experimental data processing requirements.

Rosser, Robert S.↗

Global Data Assembly Center (GDAC) Report to the GHRSST Science Team

In 2012-2013 the Global Data Assembly Center (GDAC) at NASA's Physical Oceanography Distributed Active Archive Center (PO.DAAC) continued its role as the primary clearinghouse and access node for operational GHRSST data streams, as well as its collaborative role with the NOAA Long Term Stewardship and Reanalysis Facility (LTSRF) for archiving. Our presentation reported on our data management activities and infrastructure improvements since the last science team meeting in 2012.

user reports↗

Contextual Active Online Model Selection with Expert Advice

How can we collect the most useful labels to learn a model selection policy, when presented with arbitrary heterogeneous data streams? In this paper, we formulate this task as a contextual active model selection problem, where at each round the learner receives an unlabeled data point along with a context. The goal is to output the best model for any given context without obtaining an excessive amount of labels. In particular, we focus on the task of selecting pre-trained classifiers, and propose a contextual active model selection algorithm (CAMS), which relies on a novel uncertainty sampling query criterion defined on a given policy class for adaptive model selection. In comparison to prior art, our algorithm does not assume a globally optimal model. We provide rigorous theoretical analysis for the regret and query complexity under both adversarial and stochastic settings. Our experiments on several benchmark classification datasets demonstrate the algorithm’s effectiveness in terms of both regret and query complexity. Notably, to achieve the same accuracy, CAMS incurs less than 10% of the label cost when compared to the best online model selection baselines on CIFAR10.

Liu, Xuefeng↗

A deep learning approach to anomaly detection in geological carbon sequestration sites using pressure measurements

Carbon capture and storage (CCS) has been extensively investigated as a potential engineering measure to reduce anthropogenic carbon emission to the atmosphere. Real-time monitoring of the safety and integrity of carbon storage reservoirs is a critical aspect of any commercial-scale CCS deployment. Pressure-based sensing is cost effective, suitable for real-time monitoring, and scalable to large monitoring networks. However, questions remain on how to best harness intelligent information from the high-frequency pressure monitoring sensors to support real-time decisions. Here this work presents a deep-learning-based framework for analyzing and detecting anomalies in pressure data streams by using a convolutional long short-term memory (ConvLSTM) neural network model, which allows for the fusion of both static and dynamic reservoir data. In ConvLSTM, the convolutional neural network (CNN) is used for spatial pattern mining and the LSTM is used for temporal pattern recognition. The performance of the ConvLSTM model for real-time anomaly detection is demonstrated using a set of pressure monitoring data collected from Cranfield, Mississippi, an active enhanced-oil-recovery field. The anomaly detection model is trained using bottom-hole pressure data acquired from the base experiment (without leak event) and then tested on pressure data collected during a series of controlled CO 2 release experiments (with artificially created leak events). Results show that the ConvLSTM neural network model successfully detected anomalies in the pressure time series obtained from the controlled release experiments. Inclusion of static information into the model further improves the robustness of ConvLSTM.

13 HYDRO ENERGY↗

The Phase-2 Upgrade of the CMS Data Acquisition

The High Luminosity LHC (HL-LHC) will start operating in 2027 after the third Long Shutdown (LS3), and is designed to provide an ultimate instantaneous luminosity of 7:5 × 10$^{34}$ cm$^{-2}$ s$^{-1}$, at the price of extreme pileup of up to 200 interactions per crossing. The number of overlapping interactions in HL-LHC collisions, their density, and the resulting intense radiation environment, warrant an almost complete upgrade of the CMS detector. The upgraded CMS detector will be read out by approximately fifty thousand highspeed front-end optical links at an unprecedented data rate of up to 80 Tb/s, for an average expected total event size of approximately 8 - 10 MB. Following the present established design, the CMS trigger and data acquisition system will continue to feature two trigger levels, with only one synchronous hardware-based Level-1 Trigger (L1), consisting of custom electronic boards and operating on dedicated data streams, and a second level, the High Level Trigger (HLT), using software algorithms running asynchronously on standard processors and making use of the full detector data to select events for offline storage and analysis. The upgraded CMS data acquisition system will collect data fragments for Level-1 accepted events from the detector back-end modules at a rate up to 750 kHz, aggregate fragments corresponding to individual Level- 1 accepts into events, and distribute them to the HLT processors where they will be filtered further. Events accepted by the HLT will be stored permanently at a rate of up to 7.5 kHz. This paper describes the baseline design of the DAQ and HLT systems for the Phase-2 of CMS.

Badaro, Gilbert↗

Towards a study protocol: A data-driven workflow to identify error sources in direct ink write mechatronics

Abstract Using Direct Ink Write (DIW) technology in a rapid and large-scale production requires reliable quality control for printed parts. Data streams generated during printing, such as print mechatronics, are massive and diverse which impedes extracting insights. In our study protocol approach, we developed a data-driven workflow to understand the behavior of sensor-measured X- and Y- axes positional errors with process parameters, such as print velocity and velocity control. We uncovered patterns showing that instantaneous changes in the velocity, when the build platform accelerates and decelerates, largely influence the positional errors, especially in the X- axis due to the hardware architecture. Since DIW systems share similar mechatronic inputs and outputs, our study protocol approach is broadly applicable and scalable across multiple systems. Graphical abstract

36 MATERIALS SCIENCE↗

A deterministic, gigabit serial timing, synchronization and data link for the RHIC LLRF

A critical capability of the new RHIC low level rf (LLRF) system is the ability to synchronize signals across multiple locations. The 'Update Link' provides this functionality. The 'Update Link' is a deterministic serial data link based on the Xilinx RocketIO protocol that is broadcast over fiber optic cable at 1 gigabit per second (Gbps). The link provides timing events and data packets as well as time stamp information for synchronizing diagnostic data from multiple sources. The new RHIC LLRF was designed to be a flexible, modular system. The system is constructed of numerous independent RF Controller chassis. To provide synchronization among all of these chassis, the Update Link system was designed. The Update Link system provides a low latency, deterministic data path to broadcast information to all receivers in the system. The Update Link system is based on a central hub, the Update Link Master (ULM), which generates the data stream that is distributed via fiber optic links. Downstream chassis have non-deterministic connections back to the ULM that allow any chassis to provide data that is broadcast globally.

43 PARTICLE ACCELERATORS↗

Apparatus and amendment of wind turbine blade impact detection and analysis

A multisensory system provides both temporal and spatial coverage capacities for auto-detection of bird collision events. The system includes an apparatus having a first circuitry to capture and store a series of images or video of a blade of a wind turbine; and a memory to store the images from the first circuitry. The apparatus also has one or more sensors to continuously sense vibration of the blade or for acoustic recordings; and a second circuitry to analyze the sensor data stream and/or the series of images or video to identify a cause of the vibration and to trigger the camera(s). A communication interface transmits data from the second circuitry to another device, wherein the second circuitry applies artificial intelligence or machine learning to control sensitivity of the one or more sensors.

Johnston, Matthew↗

Real-time Anomaly Detection for Liquid Argon Time Projection Chambers

We present a real-time anomaly detection framework for liquid argon time projection chambers (LArTPCs), targeting applications in particle physics experiments such as the Short Baseline Near Detector (SBND) or the future Deep Underground Neutrino Experiment (DUNE). These experiments employ detectors that generate and stream high-resolution but sparse images of neutrino and other particle interactions. Our approach utilizes anomaly detection with autoencoders, compressed through knowledge distillation (KD), to enable the detection of anomalous signals in the data through efficient inference on resource-constrained hardware. The framework is targeted for deployment on computing platforms equipped with field-programmable gate arrays (FPGAs), GPUs, or CPUs, allowing low-latency selection of relevant activity directly from the raw detector data stream. We demonstrate that our approach is suitable for the detection and localization of anomalously "high-multiplicity" activity, and outline promising applications for LArTPC online data filtering and triggering.

FOS: Physical sciences↗

Machine Learning Enabled Sensor Fusion for In-Situ Defect Detection in Laser Powder Bed Fusion

Laser Powder Bed Fusion (L-PBF) Additive Manufacturing (AM) is among the metal 3D printing technologies most broadly adopted by the manufacturing industry. The current industry qualification paradigm for critical-application L-PBF parts relies heavily on expensive non-destructive inspection techniques such as X-Ray Computed Tomography (XCT), which significantly limits the use-cases of L-PBF. In situ monitoring of the process promises a less expensive alternative to ex situ testing, but existing sensor technologies and data analysis techniques struggle to detect sub-surface flaws (e.g., porosity and cracking) on production-scale L-PBF printers. RTX Technologies Research Center (RTRC) has licensed ORNL’s Peregrine software package – a printer- and camera-agnostic data analytics tool designed specifically for detecting process anomalies using in situ data collected during powder bed printing. The goal of this project was to feed temporally rich, multi-modal sensor data, including visible light, integrated near infrared (NIR), and spatially mapped co-axial melt pool thermal emission data into Peregrine to enable detection of subsurface flaws. XCT data was used as ground truth training data to allow Peregrine’s deep learning algorithms to recognize anomalies in these complex data streams in both test artifacts and industrially relevant geometries. Completion of this program has seen the successful implementation of multi-modal, multi-layer sensor data footprints for training of machine learning models in Peregrine. Flaws detected in XCT data have been successfully detected directly from this in situ data footprint, and initial analyses of the in situ probability-of-detection has been conducted, showing performance levels commensurate with traditional non-destructive evaluation (NDE) methods. The in situ monitoring methodology was then applied to an industrially relevant component that was using post-build NDE, highlighting the utility of the proposed method for hard-to-inspect AM components. As a direct result of this program, two journal manuscripts [1], [2] have been published in Additive Manufacturing, with additional manuscripts planned following program completion.

36 MATERIALS SCIENCE↗

Measurement of charged-current $\nu_{\mu}$ and $\nu_e$ interactions with wire-cell in MicroBooNE towards a search for low-energy $\nu_e$ excess

This technote summarizes the existing work in searching for $\nu_e$ low-energy excess (eLEE) in MicroBooNE Booster Neutrino Beam (BNB) data stream based on the Wire-Cell event reconstruction paradigm. The charged-current $\nu_{\mu}$ and $\nu_e$ events are selected from the 5.3e19 POT data from the BNB beam and 2.06e20 POT data from Neutrinos at the Main Injector (NuMI) beam. The charged-current $\nu_e$ selection results from the BNB data that are sensitive to the eLEE search are not included. Various comparisons between data and Monte Carlo predictions are performed to validate the overall model and demonstrate the power of the analysis techniques.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Solar maximum ultraviolet spectrometer and polarimeter

The objectives of the UVSP experiment are to study solar ultraviolet radiations, particularly from flares and active regions, and to measure constituents in the terrestrial atmosphere by the extinction of sunlight at satellite dawn and dusk. The instrument is designed to observe the Sun at a variety of spectral and spatial resolutions in the range from 1150 to 3600 A. A Gregorian telescope with effective focal length of 1.8 m is used to feed a 1 m Ebert-Fastie spectrometer. A polarimeter containing rotatable magnesium fluoride waveplates is included behind the spectrometer entrance slit and will allow all four Stokes parameters to be determined. Velocities on the Sun can also be measured. The instrument is controlled by a computer which can interact with the data stream to modify the observing program. The observing modes, including rasters, spectral scans, velocity measurements, and polarimetry, are also described along with plans for mission operations, data handling, and analysis of the observations.

Tandberg-Hanssen, E.↗

Low Latency DESDynI Data Products for Disaster Response, Resource Management and Other Applications

We are developing onboard processor technology targeted at the L-band SAR instrument onboard the planned DESDynI mission to enable formation of SAR images onboard opening possibilities for near-real-time data products to augment full data streams. Several image processing and/or interpretation techniques are being explored as possible direct-broadcast products for use by agencies in need of low-latency data, responsible for disaster mitigation and assessment, resource management, agricultural development, shipping, etc. Data collected through UAVSAR (L-band) serves as surrogate to the future DESDynI instrument. We have explored surface water extent as a tool for flooding response, and disturbance images on polarimetric backscatter of repeat pass imagery potentially useful for structural collapse (earthquake), mud/land/debris-slides etc. We have also explored building vegetation and snow/ice classifiers, via support vector machines utilizing quad-pol backscatter, cross-pol phase, and a number of derivatives (radar vegetation index, dielectric estimates, etc.). We share our qualitative and quantitative results thus far.

applications↗

Generation of Land Surface Reflectance with Combined Geo-KOMPSAT-2A AMI and Himawari 8 AHI Observations

The latest generation of geostationary satellites has opened a new era of Earth observations with unprecedented spatiotemporal resolution and spectral range. Together with GOES 16/17 ABI, FY4-A AGRI, and Himawari-8 AHI, a new Korean geostationary satellite (Geo-KOMPSAT-2A AMI) has operationally collected a full-disk image in 16 channels every ten minutes since July 2019, allowing diurnal land surface monitoring over a large proportion of Asia and all of Oceania. Retrieving accurate surface reflectance (SR) over land from GK-2A/AMI is a challenging but high priority objective. One of the challenges is the absence of a spectral band in the 2.2 m SWIR range from AMI, which is required by many atmospheric correction algorithms to retrieve atmospheric aerosol properties. To remedy this issue, we adopt a strategy that combines concurrent GK-2A/AMI and Himawari 8/AHI observations in order to derive AMI SR. We have adapted the NASA Multi-Angle Implementation of Atmospheric Correction (MAIAC) algorithm to process the data stream from Himawari 8/AHI. The advantages of the MAIAC algorithm is its capability to exploit the high temporal frequency and varying illumination geometry of the geostationary data for advanced cloud/snow detection, aerosol retrieval, and characterization of surface bidirectional reflectance properties. Leveraging the similarities of spectral bands and the sun-target-sensor geometry between AMI and AHI, we are able to create denser time series of observations and enhanced BRDF samples over most of the spatial coverage of AMI (and AHI). The combined stereo-type observations not only help derive SR for AMI but also enhance retrievals of the corresponding AHI surface products. We evaluate the resulting AMI SR using ground (AERONET) observations and corresponding MODIS products. Further, we discuss potential challenges in utilizing the geostationary satellite data for land surface monitoring.

geostationary satellite↗

Actor-based Runtime Verification with MESA

This work presents a runtime verification approach implemented in the tool MESA (MEssage-based System Analysis) which allows for using concurrent monitors to check for properties specified in data parameterized temporal logic and state machines. The tool is implemented as an internal Scala DSL. We employ the actor programming model to implement MESA where monitors are captured by concurrent actors that communicate via messaging. The paper presents a case study in which MESA is used to effectively monitor a large number of flights from live US airspace data streams. We also perform an empirical study by conducting experiments using monitoring systems with different numbers of concurrent monitors and different layers of indexing on the data contained in events. The paper describes the experiments, evaluates the results, and discusses challenges faced during the study. The evaluation shows the value of combining concurrency with indexing to handle data rich events.

runtime verification↗

ESnet/JLab FPGA Accelerated Transport

To increase the science rate for high data rates/volumes, Thomas Jefferson National Accelerator Facility (JLab) has partnered with Energy Sciences Network (ESnet) to define an edge to data center traffic shaping / steering transport capability featuring data event aware network shaping and forwarding. The keystone of this ESnet+JLab FPGA Accelerated Transport (EJFAT) is the joint development of an AI/ML directed dynamic compute work Load Balancer (LB) of UDP streamed data. The LB is a suite consisting of a Field Programmable Gate Array (FPGA) executing the dynamically configurable, low fixed latency LB data plane featuring real-time packet redirection and high throughput, and a control plane running on the FPGA host computer that monitors network and compute farm telemetry in order to make dynamic AI/ML guided decisions for destination compute host redirection/load balancing and destination resource provisioning. The LB provides for three-tier horizontal scaling across LB suites, core compute hosts, and CPUs within a host. The LB effectively provides seamless integration of edge/core computing to support direct experimental data processing for immediate use by JLab science programs and others such as the EIC as well as data centers of the future requiring high throughput and low latency for both hot and cooled data for both running experiment data acquisition systems and data center use cases.

97 MATHEMATICS AND COMPUTING↗

An Event Detection Framework for Virtual Observation System: Anomaly Identification for an ACME Land Simulation

Based on previous work on in-situ data transfer infrastructure and compiler-based software analysis, we have designed a virtual observation system for real time computer simulations. This paper presents an event detection framework for a virtual observation system. By using signal processing and detection approaches to the memory-based data streams, this framework can be reconfigured to capture high-frequency events and low-frequency events. These approaches used in the framework can dramatically reduce the data transfer needed for in-situ data analysis (between distributed computing nodes or between the CPU/GPU nodes). In the paper, we also use a terrestrial ecosystem system simulation within the Earth System Model to demonstrate the practical values of this effort.

Yao, Cindy↗