Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “streaming algorithms”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records

Implementation of a Multichannel Serial Data Streaming Algorithm using the Xilinx Serial RapidIO Solution

In the current world of applications that use reconfigurable technology implemented on field programmable gate arrays (FPGAs), there is a need for flexible architectures that can grow as the systems evolve. A project has limited resources and a fixed set of requirements that development efforts are tasked to meet. Designers must develop robust solutions that practically meet the current customer demands and also have the ability to grow for future performance. This paper describes the development of a high speed serial data streaming algorithm that allows for transmission of multiple data channels over a single serial link. The technique has the ability to change to meet new applications developed for future design considerations. This approach uses the Xilinx Serial RapidIO LOGICORE Solution to implement a flexible infrastructure to meet the current project requirements with the ability to adapt future system designs.

RapidIO↗

Final Report: Sampling-Based Algorithms for Estimating Structure in Big Data.

The purpose of this project was to develop sampling-based algorithms to discover hidden struc- ture in massive data sets. Inferring structure in large data sets is an increasingly common task in many critical national security applications. These data sets come from myriad sources, such as network traffic, sensor data, and data generated by large-scale simulations. They are often so large that traditional data mining techniques are time consuming or even infeasible. To address this problem, we focus on a class of algorithms that do not compute an exact answer, but instead use sampling to compute an approximate answer using fewer resources. The particular class of algorithms that we focus on are streaming algorithms , so called because they are designed to handle high-throughput streams of data. Streaming algorithms have only a small amount of working storage - much less than the size of the full data stream - so they must necessarily use sampling to approximate the correct answer. We present two results: * A streaming algorithm called HyperHeadTail , that estimates the degree distribution of a graph (i.e., the distribution of the number of connections for each node in a network). The degree distribution is a fundamental graph property, but prior work on estimating the degree distribution in a streaming setting was impractical for many real-world application. We improve upon prior work by developing an algorithm that can handle streams with repeated edges, and graph structures that evolve over time. * An algorithm for the task of maintaining a weighted subsample of items in a stream, when the items must be sampled according to their weight, and the weights are dynamically changing. To our knowledge, this is the first such algorithm designed for dynamically evolving weights. We expect it may be useful as a building block for other streaming algorithms on dynamic data sets.

97 MATHEMATICS AND COMPUTING↗

Fast shared-memory streaming multilevel graph partitioning

In this report we show that a fast parallel graph partitioner can benefit many applications by reducing data transfers. The online methods for partitioning graphs have to be fast and they often rely on simple one-pass streaming algorithms, while the offline methods for partitioning graphs contain more involved algorithms and the most successful methods in this category belong to the multilevel approaches. In this work, we assess the feasibility of using streaming graph partitioning algorithms within the multilevel framework. Our end goal is to come up with a fast parallel offline multilevel partitioner that can produce competitive cutsize quality. We rely on a simple but fast and flexible streaming algorithm throughout the entire multilevel framework. This streaming algorithm serves multiple purposes in the partitioning process: a clustering algorithm in the coarsening, an effective algorithm for the initial partitioning, and a fast refinement algorithm in the uncoarsening. Its simple nature also lends itself easily for parallelization. The experiments on various graphs show that our approach is on the average up to 5.1x faster than the multi-threaded MeTiS, which comes at the expense of only 2x worse cutsize.

97 MATHEMATICS AND COMPUTING↗

Streaming Solutions for Fine-Grained Network Traffic Measurements and Analysis

Online network traffic measurements and analysis is critical for detecting and preventing any real-time anomalies in the network. We propose, implement, and evaluate an online, adaptive measurement platform, which utilizes real-time traffic analysis results to refine subsequent traffic measurements. Central to our solution is the concept of Multi-Resolution Tiling (MRT), a heuristic approach that performs sequential analysis of traffic data to zoom into traffic subregions of interest. However, MRT is sensitive to transient traffic spikes. In this paper, we propose three novel traffic streaming algorithms that overcome the limitations of MRT and can cater to varying degrees of computational and storage budgets, detection latency, and accuracy of query response. We evaluate our streaming algorithms on a highly parallel and programmable hardware as well as a traditional software-based platforms. The algorithms demonstrate significant accuracy improvement over MRT in detecting anomalies consisting of synthetic hard-to-track elephant flows and global icebergs. Our proposed algorithms maintain the worst-case complexities of the MRT while incurring only a moderate increase in average resource utilization.

97 MATHEMATICS AND COMPUTING↗

HyperHeadTail v. 1.0

HyperHeadTail is a streaming algorithm for estimating the degree distribution of a graph from a stream of edges using very small storage space. Our algorithm builds upon the HeadTail algorithm of Simpson, Comandur, and McGregor [14], but extends it to handle repeat edges and variable-length sliding time windows, making it suitable for application to real-world streams such as those generated by network traffic or other communication networks.

Stolman, Andrew↗

Intercomparison and improvement of two-stream shortwave radiative transfer schemes in Earth system models for a unified treatment of cryospheric surfaces

Abstract. Snow is an important climate regulator because it greatly increases the surface albedo of middle and high latitudes of the Earth. Earth system models (ESMs) often adopt two-stream approximations with different radiative transfer techniques, the same snow therefore has different solar radiative properties depending whether it is on land or on sea ice. Here we intercompare three two-stream algorithms widely used in snow models, improve their predictions at large zenith angles, and introduce a hybrid model suitable for all cryospheric surfaces in ESMs. The algorithms are those employed by the SNow ICe and Aerosol Radiative (SNICAR) module used in land models, dEdd–AD used in Icepack, the column physics used in the Los Alamos sea ice model CICE and MPAS-Seaice, and a two-stream discrete-ordinate (2SD) model. Compared with a 16-stream benchmark model, the errors in snow visible albedo for a direct-incident beam from all three two-stream models are small (<±0.005) and increase as snow shallows, especially for aged snow. The errors in direct near-infrared (near-IR) albedo are small (<±0.005) for solar zenith angles θ<75∘, and increase as θ increases. For diffuse incidence under cloudy skies, dEdd–AD produces the most accurate snow albedo for both visible and near-IR (<±0.0002) with the lowest underestimate (−0.01) for melting thin snow. SNICAR performs similarly to dEdd–AD for visible albedos, with a slightly larger underestimate (−0.02), while it overestimates the near-IR albedo by an order of magnitude more (up to 0.04). 2SD overestimates both visible and near-IR albedo by up to 0.03. We develop a new parameterization that adjusts the underestimated direct near-IR albedo and overestimated direct near-IR heating persistent across all two-stream models for θ>75∘. These results are incorporated in a hybrid model SNICAR-AD, which can now serve as a unified solar radiative transfer model for snow in ESM land, land ice, and sea ice components.

58 GEOSCIENCES↗