Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Streaming Algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
NASA NTRS2016

Implementation of a Multichannel Serial Data Streaming Algorithm using the Xilinx Serial RapidIO Solution

In the current world of applications that use reconfigurable technology implemented on field programmable gate arrays (FPGAs), there is a need for flexible architectures that can grow as the systems evolve. A project has limited resources and a fixed set of requirements that development efforts are tasked to meet. Designers must develop robust solutions that practically meet the current customer demands and also have the ability to grow for future performance. This paper describes the development of a high speed serial data streaming algorithm that allows for transmission of multiple data channels over a single serial link. The technique has the ability to change to meet new applications developed for future design considerations. This approach uses the Xilinx Serial RapidIO LOGICORE Solution to implement a flexible infrastructure to meet the current project requirements with the ability to adapt future system designs.

RapidIO↗
NASA NTRS1986

Heating rates in tropical anvils

An IR radiative transfer model for cirrus clouds was developed on the basis of data acquired with a U-2 aircraft. The emission and scattering of radiation was expressed with a two-stream algorithm that considered the cloud in 10 layers. Exponential sums were employed to quantify gaseous absorption by H2O, CO2 and O3 over the IR spectrum of 4.5-250 microns. Ice water content had the greatest impact on radiative heating of the high altitude anvils, although the vertical extent of the anvil and the cloudiness of the underlying atmosphere were also important.

Ackerman, T. P.↗
NASA NTRS1987

New cellular automaton model for magnetohydrodynamics

A new type of two-dimensional cellular automation method is introduced for computation of magnetohydrodynamic fluid systems. Particle population is described by a 36-component tensor referred to a hexagonal lattice. By appropriate choice of the coefficients that control the modified streaming algorithm and the definition of the macroscopic fields, it is possible to compute both Lorentz-force and magnetic-induction effects. The method is local in the microscopic space and therefore suited to massively parallel computations.

Chen, Hudong↗
NASA NTRS1993

Three dimensional modeling of cirrus during the 1991 FIRE IFO 2: Detailed process study

A three-dimensional model of cirrus cloud formation and evolution, including microphysical, dynamical, and radiative processes, was used to simulate cirrus observed in the FIRE Phase 2 Cirrus field program (13 Nov. - 7 Dec. 1991). Sulfate aerosols, solution drops, ice crystals, and water vapor are all treated as interactive elements in the model. Ice crystal size distributions are fully resolved based on calculations of homogeneous freezing of solution drops, growth by water vapor deposition, evaporation, aggregation, and vertical transport. Visible and infrared radiative fluxes, and radiative heating rates are calculated using the two-stream algorithm described by Toon et al. Wind velocities, diffusion coefficients, and temperatures were taken from the MAPS analyses and the MM4 mesoscale model simulations. Within the model, moisture is transported and converted to liquid or vapor by the microphysical processes. The simulated cloud bulk and microphysical properties are shown in detail for the Nov. 26 and Dec. 5 case studies. Comparisons with lidar, radar, and in situ data are used to determine how well the simulations reproduced the observed cirrus. The roles played by various processes in the model are described in detail. The potential modes of nucleation are evaluated, and the importance of small-scale variations in temperature and humidity are discussed. The importance of competing ice crystal growth mechanisms (water vapor deposition and aggregation) are evaluated based on model simulations. Finally, the importance of ice crystal shape for crystal growth and vertical transport of ice are discussed.

Jensen, Eric J.↗
NASA NTRS1985

Study of one- and two-dimensional filtering and deconvolution algorithms for a streaming array computer

The development of vector processing computers of the streaming array architectures has made possible a dramatic decrease in the time required for the solution of problems, i.e., for those algorithms which readily lend themselves to sequential operations on long vectors. There has concurrently been rapid growth in the applications of the techniques generally known as mathematical digital filtering, also called signal analysis, digital signal processing, time series analysis, and digital image processing. The best known applications of these techniques is to seismic data and two-dimensional images, data types consisting of very large collections of numbers. A major limitation for these cases is the size and speed of the computer available. Therefore, a move to the streaming array architecture can result in marked improvement in the data analysis techniques which may be employed.

Ioup, G. E.↗
DOE OSTI2024

Streaming Matching and Edge Cover in Practice

Graph algorithms with polynomial space and time requirements often become infeasible for massive graphs with billions of edges or more. State-of-the-art approaches therefore employ approximate serial, parallel, and distributed algorithms to tackle these challenges. However, such approaches require storing the entire graph in memory and thus need access to costly computing resources such as clusters and supercomputers. In this paper, we present practical streaming approaches for solving massive graph problems using limited memory for two prototypical graph problems: maximum weighted matching and minimum weighted edge cover. For matching, we conduct a thorough computational study on two of the semi-streaming algorithms including a recent breakthrough result that achieves a $1/(2+\varepsilon)$-approximation of the weight while using $O( n \log W /\epsilon)$ memory (here $n$ is the number of vertices and $W$ is the maximum edge weight), designed by Paz and Schwartzman [SODA, 2017]. Empirically, we show that the semi-streaming algorithms produce matchings whose weight is close to the best $1/2$-approximate offline algorithm while requiring less time and an order-of-magnitude less memory. For minimum weighted edge cover, we develop three novel semi-streaming algorithms. Two of these algorithms require a single pass through the input graph, require $O(n \log n)$ memory, and provide a 2-approximation guarantee on the objective. We also leverage a relationship between approximate maximum weighted matching and approximate minimum weighted edge cover to develop a two-pass $3/2+\epsilon$-approximate algorithm with the memory requirement of Paz and Schwartzman's semi-streaming matching algorithm. These streaming approaches are compared against the state-of-the-art 3/2-approximate offline algorithm. The semi-streaming matching and the novel edge cover algorithms proposed in this paper can process graphs with several billions of edges in under 30 minutes using 6 GB of memory, which is at least an order of magnitude improvement from the offline (non-streaming) algorithms. For the largest graph, the best alternative offline parallel approximation algorithm (GPA+ROMA) could not finish in three hours even while employing hundreds of processors and 1 TB of memory. We also demonstrate an application of the semi-streaming algorithm by computing a matching using linearly bounded memory on item intersection graphs derived from three machine learning datasets, whereas the existing offline algorithms could not complete on one of these datasets since their memory requirements exceeded 1TB.

Ferdous, S M.↗
DOE OSTI2025

Streaming Compression of Scientific Data via Weak-SINDy

Here, in this paper, a streaming weak-SINDy algorithm is developed specifically for compressing streaming scientific data. The production of scientific data, either via simulation or experiments, is undergoing a stage of exponential growth, which makes data compression important and often necessary for storing and utilizing large scientific data sets. As opposed to classical “offline” compression algorithms that perform compression on a readily available data set, streaming compression algorithms compress data “online” while the data generated from simulation or experiments is still flowing through the system. This feature makes streaming compression algorithms well suited for scientific data compression, where storing the full data set offline is often infeasible. This work proposes a new streaming compression algorithm, streaming weak-SINDy, which takes advantage of the underlying data characteristics during compression. The streaming weak-SINDy algorithm constructs feature matrices and target vectors in the online stage via a streaming integration method in a memory efficient manner. The feature matrices and target vectors are then used in the offline stage to build a model through a regression process that aims to recover equations that govern the evolution of the data. For compressing high-dimensional streaming data, we adopt a streaming proper orthogonal decomposition (POD) process to reduce the data dimension and then use the streaming weak-SINDy algorithm to compress the temporal data of the POD expansion. We propose modifications to the streaming weak-SINDy algorithm to accommodate the dynamically updated POD basis. By combining the built model from the streaming weak-SINDy algorithm and a small amount of data samples, the full data flow could be reconstructed accurately at a low memory cost, as shown in the numerical tests.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Real Time implementation of Artificial Intelligence compression algorithm for High-Speed Streaming Readout signals

The new generation of high-energy physics experiments plans to acquire data in streaming mode. With this approach, it is possible to access the information of the whole detector (organized in time slices) for optimal and lossless triggering of data acquisitions. With this approach, data rates, especially in large detectors, are often very high, and the network is likely to be the bottleneck for the entire Streaming Read Out system. The aim of this work is to study the implementation of a lossy compression algorithm based on Artificial Intelligence: an Autoencoder. With Machine Learning it is possible to achieve a high compression ratio and fast inference time with only a small degradation of the signals, almost negligible for the specific application. This work explores different configurations of the Autoencoder and the implementation on different hardware. Different Autoencoder configurations are explored to find the best trade-off between compression ratio and reconstruction loss, both for signals and energy spectrum. Different hardware implementations are also explored to find the best platform to achieve real-time performance for the specific application.

Rossi, Fabio (ORCID:0009000385713885)↗
NASA NTRS1990

Edge detection applied to SST fields

An algorithm designed to detect fronts automatically in satellite-derived sea-surface temperature (SST) fields is presented. The algorithm is operated at different levels to detect and differentiate between false and true edges. For purposes of comparison, the algorithm is applied to a test set of 98 SST images to detect the northern edge of the Gulf Stream. The algorithm successfully detected valid temperature fronts and ignored false edges, and also produced statistics about the temperature fronts that are useful in the subsequent analysis of these fronts. It is assumed that the algorithm performs equally well on other SST fronts such as those associated with rings, the subtropical convergence, or the shelf/slope fronts.

Cayula, Jean-Francois↗
NASA NTRS1980

Algorithm for calculating turbine cooling flow and the resulting decrease in turbine efficiency

An algorithm is presented for calculating both the quantity of compressor bleed flow required to cool the turbine and the decrease in turbine efficiency caused by the injection of cooling air into the gas stream. The algorithm, which is intended for an axial flow, air routine in a properly written thermodynamic cycle code. Ten different cooling configurations are available for each row of cooled airfoils in the turbine. Results from the algorithm are substantiated by comparison with flows predicted by major engine manufacturers for given bulk metal temperatures and given cooling configurations. A list of definitions for the terms in the subroutine is presented.

Gauntner, J. W.↗
NASA NTRS1997

Departure Trajectory Synthesis and the Intercept Problem

Two areas of the departure problem in air traffic control are discussed. The first topic is the generation of climb-out trajectories to a fix. The trajectories would be utilized by a scheduling algorithm to allocate runways, sequence the proposed departures, and assign a departure time. The second area is concerned with finding horizontal trajectories to merge aircraft from the TRACON to an open slot in the en-route environment. Solutions are presented for the intercept problem for two cases: (1) the aircraft is traveling at the speed of the aircraft in the jetway; (2) the merging aircraft has to accelerate to reach the speed of the aircraft in the en-route stream. An algorithm is given regarding the computation of a solution for the latter case. For the former, a set of equations is given that allows us to numerically solve for the coordinate where the merge will occur.

Bolender, Michael A.↗
DOE OSTI2026

Evaluating lightweight unsupervised online IDS for masquerade attacks in CAN

Vehicular controller area networks (CANs) are susceptible to masquerade attacks by malicious adversaries. In masquerade attacks, adversaries silence a targeted ID and then send malicious frames with forged content at the expected timing of benign frames. As masquerade attacks could seriously harm vehicle functionality and are the stealthiest attacks to detect in CAN, recent work has devoted attention to compare frameworks for detecting masquerade attacks in CAN. However, most existing works report offline evaluations using CAN logs already collected using simulations that do not comply with the domain’s real-time constraints. Here we contribute to advance the state of the art by presenting a comparative evaluation of four different non-deep learning (DL)-based unsupervised online intrusion detection systems (IDS) for masquerade attacks in CAN. Our approach differs from existing comparative evaluations in that we analyze the effect of controlling streaming data conditions in a sliding window setting. In doing so, we use realistic masquerade attacks being replayed from the ROAD dataset. We show that although evaluated IDS are not effective at detecting every attack type, the method that relies on detecting changes in the hierarchical structure of clusters of time series produces the best results at the expense of higher computational overhead. We discuss limitations, open challenges, and how the evaluated methods can be used for practical unsupervised online CAN IDS for masquerade attacks.

Anomaly detection↗
DOE OSTI2024

Refining HPCToolkit for application performance analysis at exascale

As part of the US Department of Energy’s Exascale Computing Project (ECP), Rice University has been refining its HPCToolkit performance tools to better support measurement and analysis of applications executing on exascale supercomputers. To efficiently collect performance measurements of GPU-accelerated applications, HPCToolkit employs novel non-blocking data structures to communicate performance measurements between tool threads and application threads. To attribute performance information in detail to source lines, loop nests, and inlined call chains, HPCToolkit performs parallel analysis of large CPU and GPU binaries involved in the execution of an exascale application to rapidly recover mappings between machine instructions and source code. To analyze terabytes of performance measurements gathered during executions at exascale, HPCToolkit employs distributed-memory parallelism, multithreading, sparse data structures, and out-of-core streaming analysis algorithms. To support interactive exploration of profiles up to terabytes in size, HPCToolkit’s hpcviewer graphical user interface uses out-of-core methods to visualize performance data. The result of these efforts is that HPCToolkit now supports collection, analysis, and presentation of profiles and traces of GPU-accelerated applications at exascale. These improvements have enabled HPCToolkit to efficiently measure, analyze and explore terabytes of performance data for executions using as many as 64K MPI ranks and 64K GPU tiles on ORNL’s Frontier supercomputer. HPCToolkit’s support for measurement and analysis of GPU-accelerated applications has been employed to study a collection of open-science applications developed as part of ECP. This paper reports on these experiences, which provided insight into opportunities for tuning applications, strengths and weaknesses of HPCToolkit itself, as well as unexpected behaviors in executions at exascale.

Adhianto, Laksono↗
NASA NTRS2021

Algorithm Theoretical Basis Document (ATBD) - Stream Stage Measurements: V2.5.1 Water Level Products from Satellite Radar Altimetry

In response to the 2018 NASA ROSES Applied Sciences/Water Resources (NASA HQ Program Official: Dr. Brad Doorn) call for proposals, the “Integration of Remotely Sensed Streamflow Data into Alaska Water Resource Management Agency Operations” project with Principal Investigator (PI) Jack Eggleston USGS, was successful, and had the ultimate goal of creating a series of remotely sensed or derived Alaska river parameters for integration into NWIS. These parameters included surface water height and average reach surface water slope (from altimetry), average reach width (from Landsat imagery), and an associated river discharge derived via theoretical means. The surface water height products were required to have both archival and near real time components, noting the availability of ~25years of potential measurements, and accepting the temporal resolution (10-35days) of the suite of radar altimeters. Each surface water level product was expected to be a continuous time series of observation with a sufficient accuracy to highlight monthly, seasonal and interannual variation. The designated set of river reaches were chosen for their geographical distribution, their reach width, and the presence of a radar altimeter mission satellite overpass. This document describes the procedure associated with the creation of these altimetric surface water level products and is relevant to product Version 2.5.1 available from the Global Water Monitor (GWM) web portal.

Altimetry↗
DOE OSTI2024

Kernelized approaches to streaming compression of scientific data

In this paper three algorithms are developed for the streaming compression of scientific data. The algorithms presented are reliant on the theory of vector-valued reproducing kernel Hilbert spaces and operator valued kernel. Further, the scientific data is modeled as a snapshot of time dependent vector field F(x, t) over a manifold M and the recovery of the data is framed as a learning problem. These processes are then appropriately modified and ana lyzed for the streaming scenario in which data is generated without the ability to revisit past entries.

97 MATHEMATICS AND COMPUTING↗