Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

Z-checker: A framework for assessing lossy compression of scientific data

Because of the vast volume of data being produced by today’s scientific simulations and experiments, lossy data compressor allowing user-controlled loss of accuracy during the compression is a relevant solution for significantly reducing the data size. However, lossy compressor developers and users are missing a tool to explore the features of scientific data sets and understand the data alteration after compression in a systematic and reliable way. To address this gap, we have designed and implemented a generic framework called Z-checker. On the one hand, Z-checker combines a battery of data analysis components for data compression. On the other hand, Z-checker is implemented as an open-source community tool to which users and developers can contribute and add new analysis components based on their additional analysis demands. In this article, we present a survey of existing lossy compressors. Then, we describe the design framework of Z-checker, in which we integrated evaluation metrics proposed in prior work as well as other analysis tools. Specifically, for lossy compressor developers, Z-checker can be used to characterize critical properties (such as entropy, distribution, power spectrum, principal component analysis, and autocorrelation) of any data set to improve compression strategies. For lossy compression users, Z-checker can detect the compression quality (compression ratio and bit rate) and provide various global distortion analysis comparing the original data with the decompressed data (peak signal-to-noise ratio, normalized mean squared error, rate–distortion, rate-compression error, spectral, distribution, and derivatives) and statistical analysis of the compression error (maximum, minimum, and average error; autocorrelation; and distribution of errors). Z-checker can perform the analysis with either coarse granularity (throughout the whole data set) or fine granularity (by user-defined blocks), such that the users and developers can select the best fit, adaptive compressors for different parts of the data set. Z-checker features a visualization interface displaying all analysis results in addition to some basic views of the data sets such as time series. To the best of our knowledge, Z-checker is the first tool designed to assess lossy compression comprehensively for scientific data sets.

97 MATHEMATICS AND COMPUTING↗

Low-Complexity Adaptive Lossless Compression of Hyperspectral Imagery

A low-complexity, adaptive predictive technique for lossless compression of hyperspectral imagery is described. This technique is designed to be suitable for implementation in hardware such as a field programmable gate array (FPGA); such an implementation could be used for high-speed compression of hyperspectral imagery onboard a spacecraft. The predictive step of the technique makes use of the sign algorithm, which is a relative of the least mean square (LMS) algorithm from the field of low-complexity adaptive filtering. The compressed data stream consists of prediction residuals encoded using a method similar to that of the JPEG-LS lossless image compression standard. Compression results are presented for several datasets including some raw Airborne Visible/ Infrared Imaging Spectrometer (AVIRIS) datasets and raw Atmospheric Infrared Sounder (AIRS) datasets. The compression effectiveness obtained with the technique is competitive with that of the best of previously described techniques with similar complexity.

lossless data compression↗

waveSZ: A Hardware-Algorithm Co-Design of Efficient Lossy Compression for Scientific Data

Error-bounded lossy compression is critical to the success of extreme-scale scientific research because of ever-increasing volumes of data produced by today's high-performance computing (HPC) applications. Not only can error-controlled lossy compressors significantly reduce the I/O and storage burden but they can retain high data fidelity for post analysis. Existing state-of-the-art lossy compressors, however, generally suffer from relatively low compression and decompression throughput (up to hundreds of megabytes per second on a single CPU core), which considerably restrict the adoption of lossy compression by many HPC applications especially those with a fairly high data production rate. In this paper, we propose a highly efficient lossy compression approach based on field programmable gate arrays (FPGAs) under the state-of-the-art lossy compression model SZ. Our contributions are fourfold. (1) We adopt a wavefront memory layout to alleviate the data dependency during the prediction for higher-dimensional predictors, such as the Lorenzo predictor. (2) We propose a co-design framework named WAVESZ based on the wavefront memory layout and the characteristics of SZ algorithm and carefully implement it by using high-level synthesis. (3) We propose a hardware-algorithm co-optimization method to improve the performance. (4) We evaluate our proposed WAVESZ on three real-world HPC simulation datasets from the Scientific Data Reduction Benchmarks and compare it with other state-of-the-art methods on both CPUs and FPGAs. Experiments show that our WAVESZ can improve SZ's compression throughput by 6.9x similar to 8.7x over the production version running on a state-of-the-art CPU and improve the compression ratio and throughput by 2.1x and 5.8x on average, respectively, respectively, compared with the state-of-the-art FPGA design.

Compression Ratio↗

Examination of uncertainties in historical UO 2 data using the flash gap technique

Historical shock compression data on porous uranium dioxide, UO 2 , reported by Goplen [Goplen, LA-04319-MS, 1970] is examined within the context of contemporary measurement and analysis techniques for the assignment of uncertainties. It is suggested that all data for initial density ρ 00 = 10.3 g/cm 3 with pressures below 93.1 GPa be omitted from further analysis due to the possibility that a strong elastic component to the shock wave polluted the Hugoniot measurement. Furthermore, extensive uncertainty bounds on the two lowest initial density samples, ρ 00 = 3.1 and 4.3 g/cm 3 , preclude meaningful interpretation of the data and should also be omitted from further analysis.

36 MATERIALS SCIENCE↗

Comparisons of theoretical limits for source coding with practical compression algorithms

The performance achieved by some specific data compression algorithms is compared with absolute limits prescribed by rate distortion theory for Gaussian sources under the mean square error distortion criterion. These results show the gains available from source coding and can be used as a reference for the evaluation of future compression schemes. Some current schemes perform well, but there is still room for improvement.

Pollara, F.↗

Wavelet compression techniques for hyperspectral data

Hyperspectral sensors are electro-optic sensors which typically operate in visible and near infrared bands. Their characteristic property is the ability to resolve a relatively large number (i.e., tens to hundreds) of contiguous spectral bands to produce a detailed profile of the electromagnetic spectrum. In contrast, multispectral sensors measure relatively few non-contiguous spectral bands. Like multispectral sensors, hyperspectral sensors are often also imaging sensors, measuring spectra over an array of spatial resolution cells. The data produced may thus be viewed as a three dimensional array of samples in which two dimensions correspond to spatial position and the third to wavelength. Because they multiply the already large storage/transmission bandwidth requirements of conventional digital images, hyperspectral sensors generate formidable torrents of data. Their fine spectral resolution typically results in high redundancy in the spectral dimension, so that hyperspectral data sets are excellent candidates for compression. Although there have been a number of studies of compression algorithms for multispectral data, we are not aware of any published results for hyperspectral data. Three algorithms for hyperspectral data compression are compared. They were selected as representatives of three major approaches for extending conventional lossy image compression techniques to hyperspectral data. The simplest approach treats the data as an ensemble of images and compresses each image independently, ignoring the correlation between spectral bands. The second approach transforms the data to decorrelate the spectral bands, and then compresses the transformed data as a set of independent images. The third approach directly generalizes two-dimensional transform coding by applying a three-dimensional transform as part of the usual transform-quantize-entropy code procedure. The algorithms studied all use the discrete wavelet transform. In the first two cases, a wavelet transform coder was used for the two-dimensional compression. The third case used a three dimensional extension of this same algorithm.

Evans, Bruce↗

Improving I/O Throughput with PRIMACY: Preconditioning ID-Mapper for Compressing Incompressibility

The ability to efficiently handle massive amounts of data is necessary for the continuing development towards exascale scientific data-mining applications and database systems. Unfortunately, recent years have shown a growing gap between the size and complexity of data produced from scientific applications and the limited I/O bandwidth available on modern high-performance computing systems. Utilizing data compression in order to lower the degree of I/O activity offers a promising means to addressing this problem. However, the standard compression algorithms previously explored for such use offer limited gains on both the end-to-end throughput and storage fronts. In this paper, we introduce an in-situ compression scheme aimed at improving end-to-end I/O throughput as well as reduction of dataset size. Our technique, PRIMACY (Preconditioning Id-MApper for Compressing incompressibility), acts as a preconditioner for standard compression libraries by modifying representation of original floating-point scientific data to increase byte-level repeatability, allowing standard loss less compressors to take advantage of their entropy-based byte-level encoding schemes. We additionally present a theoretical model for compression efficiency in high-performance computing environments and evaluate the efficiency of our approach via comparative analysis. Based on our evaluations on 20 real-world scientific datasets, PRIMACY achieved up to 38% and 22% improvements upon standard end-to-end write and read throughputs respectively in addition to a 25% increase in compression ratios paired with 3-to-4-fold improvement in both compression and decompression throughput over general purpose compressors.

Computer Science↗

Shifting Between Compute and Memory Bounds: A Compression-Enabled Roofline Model

In the evolving landscape of high-performance computing, especially to fight the end of Moore’s Law and Dennard’s Scaling, the ability to shift between compute-bound and memory-bound states is critical for enhancing adaptability and flexibility to diverse system and domain-specific architectures. Such capability is vital for optimizing performance across distinguished hardware configurations, such as accelerators, memory hierarchies, and cache systems. Despite that ad hoc optimization techniques, such as compressed/approximate computation, have been enabled for compute-/data-intensive computing for improved performance in distinct hardware settings, there lacks an understanding of 1) the rational behind performance improvement; 2) capability of different optimizations; 3) what optimization to respond to specific computational and memory demands. This work proposes a compression-enabled roofline model to facilitate this adaptability with data compression techniques to balance and transform between computational and memory demands. This model enables applications to adjust in response to the specific strengths and limitations of the underlying hardware and system to optimize resource utilization. The effectiveness of this approach is demonstrated with matrix multiplication kernels on different input sizes, with turning on/off various compression techniques, including 1) low-precision floating point; 2) sparse matrix formulation; and 3) compressed arrays with ZFP. By reducing memory transfer volumes and cache misses and increasing data locality and computational intensity through compression, the specific roofline model can transform between compute and memory bounds to align more efficiently with system capabilities. This advancement not only improves overall performance but also maximizes adaptability in diverse computing environments.

Naraparaju, Ramasoumya [University of Washington]↗

Time varying, multivariate volume data reduction

Large-scale supercomputing is revolutionizing the way science is conducted. A growing challenge, however, is understanding the massive quantities of data produced by large-scale simulations. The data, typically time-varying, multivariate, and volumetric, can occupy from hundreds of gigabytes to several terabytes of storage space. Transferring and processing volume data of such sizes is prohibitively expensive and resource intensive. Although it may not be possible to entirely alleviate these problems, data compression should be considered as part of a viable solution, especially when the primary means of data analysis is volume rendering. In this paper we present our study of multivariate compression, which exploits correlations among related variables, for volume rendering. Two configurations for multidimensional compression based on vector quantization are examined. We emphasize quality reconstruction and interactive rendering, which leads us to a solution using graphics hardware to perform on-the-fly decompression during rendering. In this paper we present a solution which addresses the need for data reduction in large supercomputing environments where data resulting from simulations occupies tremendous amounts of storage. Our solution employs a lossy encoding scheme to acrueve data reduction with several options in terms of rate-distortion behavior. We focus on encoding of multiple variables together, with optional compression in space and time. The compressed volumes can be rendered directly with commodity graphics cards at interactive frame rates and rendering quality similar to that of static volume renderers. Compression results using a multivariate time-varying data set indicate that encoding multiple variables results in acceptable performance in the case of spatial and temporal encoding as compared to independent compression of variables. The relative performance of spatial vs. temporal compression is data dependent, although temporal compression has the advantage of offering smooth animations, while spatial compression can handle volumes of larger dimensions.

97 MATHEMATICS AND COMPUTING↗

Advances in fast 2D camera data handling and analysis on NSTX

The use of fast 2D cameras on NSTX continues to grow. There are 6 cameras with the capability of taking up to 1–2 gigabytes (GBs) of data apiece during each plasma shot on the National Spherical Torus Experiment (NSTX). Efficient storage and retrieval of this data remains a challenge. Performance comparisons are presented for reading data stored in MDSplus, using both compressed data and segmented records, and direct access I/O with different read sizes. Encouragingly, fast 2D camera data provides considerable insight into plasma complexities, such as small-scale turbulence and particle transport. The last part of this paper is an example of more recent uses: dual cameras looking at the same region of the plasma from different angles, which can provide trajectories of incandescent particles in 3D. A laboratory simulation of the 3D trajectories is presented, as well as corresponding data from NSTX plasma where glowing dust particles can be followed.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Goated: goal-oriented tensor decompositions in python

SAND2026-20464O Goated performs goal-oriented tensor decompositions in Python, enabling efficient compression of multi-dimensional simulation data. It extends common tensor decomposition methods by incorporating domain-specific knowledge, such as conservation laws in physics, through a penalty term in the optimization process. This approach improves data compression and modeling accuracy across various applications, including physics simulations, by using specialized algorithms and structure-aware subroutines to accelerate solver performance. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-NA0003525.

SciDAC↗

Advanced imaging communication system

Key elements of system are imaging and nonimaging sensors, data compressor/decompressor, interleaved Reed-Solomon block coder, convolutional-encoded/Viterbi-decoded telemetry channel, and Reed-Solomon decoding. Data compression provides efficient representation of sensor data, and channel coding improves reliability of data transmission.

Hilbert, E. E.↗

Telemetry and data relay for manned space programs

Previously and currently planned methods of transferring or relaying telemetered information from a remote receiving station to the Mission Control Center, at the National Aeronautics and Space Administration Lyndon B. Johnson Space Center, are discussed. The data flow techniques of the Apollo Program (fixed formats), the Skylab Program (data compression), and the Space Shuttle Program (data throughput) are defined and analyzed according to their advantages and deficiencies from an operations viewpoint. Excluded from the data flow technique discussions are command data, tracking data, television data, and Teletype and Datafax messages. Voice transmissions are addressed in the Space Shuttle Program discussion because they form a part of the telemetry data stream. The tradeoffs between actual data flow requirements and existing communications limitations are addressed for each of these programs. Finally, a short discussion of current problem areas in the planned ground data transmission system for the Space Shuttle Program is presented.

Anson, K. W.↗

Phase Calibration Of Radar Polarimetric Data

Technique for phase calibration of data acquired by airborne imaging radar polarimeter based on extraction of calibration parameters from data themselves. Enables use of data-compression technique to reduce volume of data in synthetic-aperture-radar correlator. Typical radar polarimeter includes transmitting and receiving channels for horizontally and vertically polarized signals. Phase delay in each channel usually known only approximately if at all. Consequently, necessary to phase-calibrate radar return signals.

Zebker, Howard A.↗

Intelligent vehicle control: Opportunities for terrestrial-space system integration

For 11 years the Department of Defense has cooperated with a diverse array of other Federal agencies including the National Institute of Standards and Technology, the Jet Propulsion Laboratory, and the Department of Energy, to develop robotics technology for unmanned ground systems. These activities have addressed control system architectures supporting sharing of tasks between the system operator and various automated subsystems, man-machine interfaces to intelligent vehicles systems, video compression supporting vehicle driving in low data rate digital communication environments, multiple simultaneous vehicle control by a single operator, path planning and retrace, and automated obstacle detection and avoidance subsystem. Performance metrics and test facilities for robotic vehicles were developed permitting objective performance assessment of a variety of operator-automated vehicle control regimes. Progress in these areas will be described in the context of robotic vehicle testbeds specifically developed for automated vehicle research. These initiatives, particularly as regards the data compression, task sharing, and automated mobility topics, also have relevance in the space environment. The intersection of technology development interests between these two communities will be discussed in this paper.

Shoemaker, Charles↗

Ocean-Science Mission Needs: Real-Time AUV Data for Command, Control, and Model Inputs

Predictive models for tides, hydrodynamics, and bio-optical properties affecting the visibility and buoyancy of coastal waters are needed to evaluate the safety of personnel and equipment engaged in maritime operations under potentially hazardous conditions. Predicted currents can be markedly different for two-layer systems affected by terrestrial runoff than for well-mixed conditions because the layering decouples the surface and bottom Ekman layers and rectifies the current response to oscillatory upwelling-and downwelling-favorable winds. Standard ocean models (e.g. Princeton Ocean Model) require initial-and boundary data on the physical and optical properties of the multilayered water column to provide accurate simulations of heat budgets and circulation. Two observational systems are designed to measure vertically structured conditions on the West Florida Shelf (WFS): a tethered buoy network and an autonomous underwater vehicle (AUV) observational system. The AUV system is described with a focus on the observational systems that challenge or limit the communications command and control network for various types of measurement programs. These include vertical oscillatory missions on shelf transects to observe the optical and hydrographic properties of the water column, and bottom-following missions for measuring the bottom albedo. Models of light propagation, absorption, and conversion to heat as well as determination of the buoyancy terms for physical models require these measurements. High data rates associated with video bottom imagery are the most challenging for the real-time, command and control communications system, but they are met through a combination of loss-less and lossy data-compression methods, depending upon the data-rate of the radio links.

Carder, Kendall L.↗

TES-8: Advanced Exo-Brake, VR and COM Experiments

The TES-8 was jettisoned from the International Space Station on January 31, 2019. As an orbital laboratory and 8th in on-going series, the design makes use of a standard set of interfaces and safety features that permit rapid re-flight. On this flight, an advanced Exo-Brake is flown with de-orbit targeting capability that will engender sample return capability from LEO platforms. A Virtual Reality data recording system uses stereo imaging and efficient data-compression with an NVIDIA GPU (Graphics Processing Unit) to permit compression and transmission of very large data files. An SDR (Software Defined Radio) will download data to the NEN (Near Earth Network) for the first time - demonstrating potential use in cis-lunar space using S-band. For the first time, a comparison will be made regarding the functionality of the Iridium and Globalstar short burst data modems - as essential communication tools for future nano-sat projects. Lastly, the 7 micro-processors and 4 cameras provide an excellent learning platform for university students and NASA young professionals.

Exo-Brake↗

Technology Directions for the 21st Century

Data compression is an important tool for reducing the bandwidth of communications systems, and thus for reducing the size, weight, and power of spacecraft systems. For data requiring lossless transmissions, including most science data from spacecraft sensors, small compression factors of two to three may be expected. Little improvement can be expected over time. For data that is suitable for lossy compression, such as video data streams, much higher compression factors can be expected, such as 100 or more. More progress can be expected in this branch of the field, since there is more hidden redundancy and many more ways to exploit that redundancy.

Crimi, Giles↗