Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3

Fast Algorithms for Scientific Data Compression

Many scientific simulations and experiments generate terabytes to petabytes of data daily, necessitating data compression techniques. Unlike video and image compression, scientists require methods that accurately preserve primary data (PD) and derived quantities of interest (QoIs). In our previous work, we demonstrated the effectiveness of hybrid compression techniques that combine machine learning with traditional approaches. This paper presents innovative computational techniques aimed at expediting the compression pipeline. Our experiments, conducted on two distinct platforms with a large-scale XGC-based fusion simulation, demonstrate that the overhead incurred by these new approaches is less than one percent of the computational resources needed for the simulation.

Banerjee, Tania↗

Reliable voltage scaled links for compressed data

Systems, apparatuses, and methods for reliably transmitting data over voltage scaled links are disclosed. A computing system includes at least first and second devices connected via a link. In one implementation, if a data block can be compressed to less than or equal to half the original size of the data block, then the data block is compressed and sent on the link in a single clock cycle rather than two clock cycles. If the data block cannot be compressed to half the original size, but if the data block can be compressed enough to include error correction code (ECC) bits without exceeding the original size, then ECC bits are added to the compressed block which is sent on the link at a reduced voltage. The ECC bits help to correct for any errors that are generated as a result of operating the link at the reduced voltage.

42 ENGINEERING↗

TuckerMPI: A Parallel C++/MPI Software Package for Large-scale Data Compression via the Tucker Tensor Decomposition

With this study, our goal is compression of massive-scale grid-structured data, such as the multi-terabyte output of a high-fidelity computational simulation. For such data sets, we have developed a new software package called TuckerMPI, a parallel C++/MPI software package for compressing distributed data. The approach is based on treating the data as a tensor, i.e., a multidimensional array, and computing its truncated Tucker decomposition, a higher-order analogue to the truncated singular value decomposition of a matrix. The result is a low-rank approximation of the original tensor-structured data. Compression efficiency is achieved by detecting latent global structure within the data, which we contrast to most compression methods that are focused on local structure. In this work, we describe TuckerMPI, our implementation of the truncated Tucker decomposition, including details of the data distribution and in-memory layouts, the parallel and serial implementations of the key kernels, and analysis of the storage, communication, and computational costs. We test the software on 4.5 and 6.7 terabyte data sets distributed across 100 s of nodes (1,000 s of MPI processes), achieving compression ratios between 100 and 200,000×, which equates to 99--99.999% compression (depending on the desired accuracy) in substantially less time than it would take to even read the same dataset from a parallel file system. Moreover, we show that our method also allows for reconstruction of partial or down-sampled data on a single node, without a parallel computer so long as the reconstructed portion is small enough to fit on a single machine, e.g., in the instance of reconstructing/visualizing a single down-sampled time step or computing summary statistics. The code is available at https://gitlab.com/tensors/TuckerMPI.

97 MATHEMATICS AND COMPUTING↗

Deep learning for in situ data compression of large turbulent flow simulations

As the size of turbulent flow simulations continues to grow, in situ data compression is becoming increasingly important for visualization, analysis, and restart checkpointing. For these applications, single-pass compression techniques with low computational and communication overhead are crucial. In this paper we present a deep-learning approach to in situ compression using an autoencoder architecture that is customized for three-dimensional turbulent flows and is well suited for contemporary heterogeneous computing resources. The autoencoder is compared against a recently introduced randomized single-pass singular value decomposition (SVD) for three different canonical turbulent flows: decaying homogeneous isotropic turbulence, a Taylor-Green vortex, and turbulent channel flow. Our proposed fully convolutional autoencoder architecture compresses turbulent flow snapshots by a factor of 64 with a single pass, allows for arbitrarily sized input fields, is cheaper to compute than the randomized single-pass SVD for typical simulation sizes, performs well on unseen flow configurations, and has been made publicly available. The results reported here show that the autoencoder dramatically outperforms a randomized single-pass SVD with similar compression ratio and yields comparable performance to a higher-rank decomposition with an order of magnitude less compression in regard to preserving a number of important statistical quantities such as turbulent kinetic energy, enstrophy, and Reynolds stresses.

97 MATHEMATICS AND COMPUTING↗

Lossy Data Compression Effects on Wall-bounded Turbulence: Bounds on Data Reduction

Postprocessing and storage of large data sets represent one of the main computational bottlenecks in computational fluid dynamics. We assume that the accuracy necessary for computation is higher than needed for postprocessing. Therefore, in the current work we assess thresholds for data reduction as required by the most common data analysis tools used in the study of fluid flow phenomena, specifically wall-bounded turbulence. These thresholds are imposed a priori by the user in L 2 -norm, and we assess a set of parameters to identify the minimum accuracy requirements. The method considered in the present work is the discrete Legendre transform (DLT), which we evaluate in the computation of turbulence statistics, spectral analysis and resilience for cases highly-sensitive to the initial conditions. Maximum acceptable compression ratios of the original data have been found to be around 97%, depending on the application purpose. Furthermore, the new method outperforms downsampling, as well as the previously explored data truncation method based on discrete Chebyshev transform (DCT).

97 MATHEMATICS AND COMPUTING↗

Strategies for on-chip digital data compression for X-ray pixel detectors

Here, the continued desire for X-ray pixel detectors with higher frame rates will stress the ability of application-specific integrated circuit (ASIC) designers to provide sufficient off-chip bandwidth to reach continuous frame rates in the 1 MHz regime. To move from the current 10 kHz to the 1 MHz frame rate regime, ASIC designers will continue to pack as many power-hungry high-speed transceivers at the periphery of the ASIC as possible. In this paper, however, we present new strategies to make the most efficient use of the off-chip bandwidth by utilizing data compression schemes for X-ray photon-counting and charge-integrating pixel detectors. In particular, we describe a novel in-pixel compression scheme that converts from analog to digital converter units to encoded photon counts near the photon Poisson noise level and achieves a compression ratio of >1.5x independent of the dataset. In addition, we describe a simple yet efficient zero-suppression compression scheme called "zeromask" (ZM) located at the ASIC's edge before streaming data off the ASIC chip. ZM achieves average compression ratios of >4x, >7x, and >8x for high-energy X-ray diffraction, ptychography, and X-ray photon correlation spectroscopy datasets, respectively. We present the conceptual designs, register-transfer level block diagrams, and the physical ASIC implementation of these compression schemes in 65 nm CMOS. When combined, these two digital compression schemes could increase the effective off-chip bandwidth by a factor of 6-12x.

47 OTHER INSTRUMENTATION↗

Compressing data for storage in cache memories in a hierarchy of cache memories

An electronic device includes at least one compression-decompression functional block and a hierarchy of cache memories with a first cache memory and a second cache memory. The at least one compression-decompression functional block receives data in an uncompressed state, compresses the data using one of a first compression or a second compression, and, after compressing the data, provides the data to the first cache memory for storage therein. When the data is retrieved from the first cache memory to be stored in the second cache memory, when the data is compressed using the first compression, the compression-decompression functional block decompresses the data to reverse effects of the first compression on the data, thereby restoring the data to the uncompressed state and provides the data compressed using the second compression or in the uncompressed state to the second cache memory for storage therein.

97 MATHEMATICS AND COMPUTING↗

Extreme data compression for the CMB

We apply the Karhunen-Loéve methods to cosmic microwave background (CMB) data sets, and show that we can recover the input cosmology and obtain the marginalized likelihoods in Λ cold dark matter cosmologies in under a minute, much faster than Markov chain Monte Carlo methods. This is achieved by forming a linear combination of the power spectra at each multipole l, and solving a system of simultaneous equations such that the Fisher matrix is locally unchanged. Instead of carrying out a full likelihood evaluation over the whole parameter space, we need evaluate the likelihood only for the parameter of interest, with the data compression effectively marginalizing over all other parameters. The weighting vectors contain insight about the physical effects of the parameters on the CMB anisotropy power spectrum C l . The shape and amplitude of these vectors give an intuitive feel for the physics of the CMB, the sensitivity of the observed spectrum to cosmological parameters, and the relative sensitivity of different experiments to cosmological parameters. We test this method on exact theory C l as well as on a Wilkinson Microwave Anisotropy Probe (WMAP)-like CMB data set generated from a random realization of a fiducial cosmology, comparing the compression results to those from a full likelihood analysis using CosmoMC. Furthermore, after showing that the method works, we apply it to the temperature power spectrum from the WMAP seven-year data release, and discuss the successes and limitations of our method as applied to a real data set.

79 ASTRONOMY AND ASTROPHYSICS↗

A General Framework for Progressive Data Compression and Retrieval

In scientific simulations, observations, and experiments, the transfer of data to and from disk and across networks has become a major bottleneck for data analysis and visualization. Compression techniques have been employed to tackle this challenge, but traditional lossy methods often demand conservative error tolerances to meet the numerical accuracy requirements of both anticipated and unknown data analysis tasks. Progressive data compression and retrieval has emerged as a promising solution, where each analysis task dictates its own accuracy needs. However, few analysis algorithms inherently support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. Here, this paper presents a framework that enables progressive-precision data queries for any data compressor or numerical representation. Our strategy hinges on a multi-component representation that successively reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented this approach with four established scientific data compressors and assessed its effectiveness using real-world data sets from the SDRBench collection. The results show that our framework competes in accuracy with the standalone compressors it is based upon. Additionally, (de)compression time is proportional to the number of components requested by the user. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.

97 MATHEMATICS AND COMPUTING↗

Data compression and covariance matrix inspection: Cosmic shear

Covariance matrices are among the most difficult pieces of end-to-end cosmological analyses. In principle, for two-point functions, each component involves a four-point function, and the resulting covariance often has hundreds of thousands of elements. We investigate various compression mechanisms capable of vastly reducing the size of the covariance matrix in the context of cosmic shear statistics. This helps identify which of its parts are most crucial to parameter estimation. We start with simple compression methods, by isolating and “removing” 200 modes associated with the lowest eigenvalues, then those with the lowest signal-to-noise ratio, before moving on to more sophisticated schemes like compression at the tomographic level and, finally, with the massively optimized parameter estimation and data compression (MOPED). We find that, while most of these approaches prove useful for a few parameters of interest, like Ω m , the simplest yield a loss of constraining power on the intrinsic alignment (IA) parameters as well as S 8 . For the case considered—cosmic shear from the first year of data from the Dark Energy Survey—only MOPED was able to replicate the original constraints in the 16-parameter space. Finally, we apply a tolerance test to the elements of the compressed covariance matrix obtained with MOPED and confirm that the IA parameter A IA is the most susceptible to inaccuracies in the covariance matrix.

79 ASTRONOMY AND ASTROPHYSICS↗

Reducing Soft-error Vulnerability of Caches using Data Compression

With ongoing chip miniaturization and voltage scaling, particle strike-induced soft errors present increasingly severe threat to the reliability of on-chip caches. In this paper, we present a technique to reduce the vulnerability of caches to soft-errors. Our technique uses data compression to reduce the number of vulnerable data bits in the cache and performs selective duplication of more critical data-bits to provide extra protection to them. Microarchitectural simulations have shown that our technique is effective in reducing architectural vulnerability factor (AVF) of the cache and outperforms another technique. For single and dual-core system configuration, the average reduction in AVF is 5.59X and 8.44X, respectively. Also, the implementation and performance overheads of our technique are minimal and it is useful for a broad range of workloads.

Mittal, Sparsh↗

Multi-level Layout Optimization for Efficient Spatio-temporal Queries on ISABELA-compressed Data

The size and scope of cutting-edge scientific simulations are growing much faster than the I/O subsystems of their runtime environments, not only making I/O the primary bottleneck, but also consuming space that pushes the storage capacities of many computing facilities. These problems are exacerbated by the need to perform data-intensive analytics applications, such as querying the dataset by variable and spatio-temporal constraints, for what current database technologies commonly build query indices of size greater than that of the raw data. To help solve these problems, we present a parallel query-processing engine that can handle both range queries and queries with spatio-temporal constraints, on B-spline compressed data with user-controlled accuracy. Our method adapts to widening gaps between computation and I/O performance by querying on compressed metadata separated into bins by variable values, utilizing Hilbert space-filling curves to optimize for spatial constraints and aggregating data access to improve locality of per-bin stored data, reducing the false positive rate and latency bound I/O operations (such as seek) substantially. We show our method to be efficient with respect to storage, computation, and I/O compared to existing database technologies optimized for query processing on scientific data.

Computer Science↗

A Tutorial on Bayesian analysis of linear shock compression data

Gas gun and other shock compression experiments often produce shock wave velocity measurements that are linearly associated with particle velocity. Traditionally, this empirical relationship is quantified with a single Hugoniot curve that is estimated using least squares regression. However, for downstream modeling and simulation tasks, it is often more useful to have multiple Hugoniot curves in the pressure–volume plane that are consistent with the data. We employ Bayesian uncertainty quantification methods as a framework for propagating measurement uncertainty through to model parameters and predictions. Specifically, this Tutorial shows how to sample multiple Hugoniot curves in the pressure–volume plane that are consistent with the shock wave-particle velocity measurements in a two-step Bayesian approach. First, we obtain an analytical expression for the posterior distribution of the linear model parameters using Bayesian linear regression. Second, we propagate samples from the posterior distribution through the Rankine–Hugoniot equations to yield Hugoniot curves in the pressure–volume plane. The procedure is demonstrated with publicly available data on argon, copper, and nickel, and compared against bootstrapping and linear regression. The Bayesian procedure is shown to be interpretable, computationally inexpensive, and less sensitive than an alternative bootstrapping approach to the removal of the point in the copper dataset that has the largest particle velocity. As a Tutorial on Bayesian methodology for the shock compression community, we provide several derivations and explanations that make this paper self-contained, and make all code and data available at github.com/llnl/BALSCD.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Synchrophasor Data Compression Under Disturbance Conditions via Cross-Entropy-Based Singular Value Decomposition

The increasing deployment of phasor measurement units and the advances of their reporting rates are challenging the present data centers in terms of storing and analyzing large-volume data. Under power system disturbance conditions, it is difficult to retain critical information while compressing the synchrophasor data effectively. This article combines the cross entropy and the singular value decomposition, proposing a novel model to compress the synchrophasor data to an extremely small size yet keep superior accuracy. The proposed model is extensively tested and compared with the state-of-the-art algorithms using the simulated and the FNET/GridEye field-collected data. The result indicates that the proposed algorithm has superior performance in compressing the data while retaining critical information under disturbance conditions.

42 ENGINEERING↗

Analytics-Driven Lossless Data Compression for Rapid In-situ Indexing, Storing, and Querying

The analysis of scientific simulations is highly data-intensive and is becoming an increasingly important challenge. Peta-scale data sets require the use of light-weight query-driven analysis methods, as opposed to heavy-weight schemes that optimize for speed at the expense of size. This paper is an attempt in the direction of query processing over losslessly compressed scientific data. We propose a co-designed double-precision compression and indexing methodology for range queries by performing unique-value-based binning on the most significant bytes of double precision data (sign, exponent, and most significant mantissa bits), and inverting the resulting metadata to produce an inverted index over a reduced data representation. Without the inverted index, our method matches or improves compression ratios over both general-purpose and floating-point compression utilities. The inverted index is light-weight, and the overall storage requirement for both reduced column and index is less than 135%, whereas existing DBMS technologies can require 200-400%. As a proof-of-concept, we evaluate univariate range queries that additionally return column values, a critical component of data analytics, against state-of-the-art bitmap indexing technology, showing multi-fold query performance improvements.

Jenkins, John↗