Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “compressing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 109 records · Page 6

Exploring Autoencoder-based Error-bounded Compression for Scientific Data

Error-bounded lossy compression is becoming an indispensable technique for the success of today's scientific projects with vast volumes of data produced during the simulations or instrument data acquisitions. Not only can it significantly reduce data size, but it also can control the compression errors based on user-specified error bounds. Autoencoder (AE) models have been widely used in image compression, but few AE-based compression approaches support error-bounding features, which are highly required by scientific applications. To address this issue, we explore using convolutional autoencoders to improve error-bounded lossy compression for scientific data, with the following three key contributions. (1) We provide an in-depth investigation of the characteristics of various autoencoder models and develop an error-bounded autoencoder-based framework in terms of the SZ model. (2) We optimize the compression quality for main stages in our designed AE-based error-bounded compression framework, fine-tuning the block sizes and latent sizes and also optimizing the compression efficiency of latent vectors. (3) We evaluate our proposed solution using five real-world scientific datasets and comparing them with six other related works. Experiments show that our solution exhibits a very competitive compression quality from among all the compressors in our tests. In absolute terms, it can obtain a much better compression quality (100%similar to 800% improvement in compression ratio with the same data distortion) compared with SZ2.1 and ZFP in cases with a high compression ratio.

Liu, Jinyang↗

Fpack and Funpack Utilities for FITS Image Compression and Uncompression

Fpack is a utility program for optimally compressing images in the FITS (Flexible Image Transport System) data format (see http://fits.gsfc.nasa.gov). The associated funpack program restores the compressed image file back to its original state (as long as a lossless compression algorithm is used). These programs may be run from the host operating system command line and are analogous to the gzip and gunzip utility programs except that they are optimized for FITS format images and offer a wider choice of compression algorithms. Fpack stores the compressed image using the FITS tiled image compression convention (see http://fits.gsfc.nasa.gov/fits_registry.html). Under this convention, the image is first divided into a user-configurable grid of rectangular tiles, and then each tile is individually compressed and stored in a variable-length array column in a FITS binary table. By default, fpack usually adopts a row-by-row tiling pattern. The FITS image header keywords remain uncompressed for fast access by FITS reading and writing software. The tiled image compression convention can in principle support any number of different compression algorithms. The fpack and funpack utilities call on routines in the CFITSIO library (http://hesarc.gsfc.nasa.gov/fitsio) to perform the actual compression and uncompression of the FITS images, which currently supports the GZIP, Rice, H-compress, and PLIO IRAF pixel list compression algorithms.

Pence, W.↗

ICER-3D Hyperspectral Image Compression Software

Software has been developed to implement the ICER-3D algorithm. ICER-3D effects progressive, three-dimensional (3D), wavelet-based compression of hyperspectral images. If a compressed data stream is truncated, the progressive nature of the algorithm enables reconstruction of hyperspectral data at fidelity commensurate with the given data volume. The ICER-3D software is capable of providing either lossless or lossy compression, and incorporates an error-containment scheme to limit the effects of data loss during transmission. The compression algorithm, which was derived from the ICER image compression algorithm, includes wavelet-transform, context-modeling, and entropy coding subalgorithms. The 3D wavelet decomposition structure used by ICER-3D exploits correlations in all three dimensions of sets of hyperspectral image data, while facilitating elimination of spectral ringing artifacts, using a technique summarized in "Improving 3D Wavelet-Based Compression of Spectral Images" (NPO-41381), NASA Tech Briefs, Vol. 33, No. 3 (March 2009), page 7a. Correlation is further exploited by a context-modeling subalgorithm, which exploits spectral dependencies in the wavelet-transformed hyperspectral data, using an algorithm that is summarized in "Context Modeler for Wavelet Compression of Hyperspectral Images" (NPO-43239), which follows this article. An important feature of ICER-3D is a scheme for limiting the adverse effects of loss of data during transmission. In this scheme, as in the similar scheme used by ICER, the spatial-frequency domain is partitioned into rectangular error-containment regions. In ICER-3D, the partitions extend through all the wavelength bands. The data in each partition are compressed independently of those in the other partitions, so that loss or corruption of data from any partition does not affect the other partitions. Furthermore, because compression is progressive within each partition, when data are lost, any data from that partition received prior to the loss can be used to reconstruct that partition at lower fidelity. By virtue of the compression improvement it achieves relative to previous means of onboard data compression, this software enables (1) increased return of hyperspectral scientific data in the presence of limits on the rates of transmission of data from spacecraft to Earth via radio communication links and/or (2) reduction in spacecraft radio-communication power and/or cost through reduction in the amounts of data required to be downlinked and stored onboard prior to downlink. The software is also suitable for compressing hyperspectral images for ground storage or archival purposes.

Xie, Hua↗

Enabling IP Header Compression in COTS Routers via Frame Relay on a Simplex Link

NASA is moving toward a networkcentric communications architecture and, in particular, is building toward use of Internet Protocol (IP) in space. The use of IP is motivated by its ubiquitous application in many communications networks and in available commercial off-the-shelf (COTS) technology. The Constellation Program intends to fit two or more voice (over IP) channels on both the forward link to, and the return link from, the Orion Crew Exploration Vehicle (CEV) during all mission phases. Efficient bandwidth utilization of the links is key for voice applications. In Voice over IP (VoIP), the IP packets are limited to small sizes to keep voice latency at a minimum. The common voice codec used in VoIP is G.729. This new algorithm produces voice audio at 8 kbps and in packets of 10-milliseconds duration. Constellation has designed the VoIP communications stack to use the combination of IP/UDP/RTP protocols where IP carries a 20-byte header, UDP (User Datagram Protocol) carries an 8-byte header, and RTP (Real Time Transport Protocol) carries a 12-byte header. The protocol headers total 40 bytes and are equal in length to a 40-byte G.729 payload, doubling the VoIP latency. Since much of the IP/UDP/RTP header information does not change from IP packet to IP packet, IP/UDP/RTP header compression can avoid transmission of much redundant data as well as reduce VoIP latency. The benefits of IP header compression are more pronounced at low data rate links such as the forward and return links during CEV launch. IP/UDP/RTP header compression codecs are well supported by many COTS routers. A common interface to the COTS routers is through frame relay. However, enabling IP header compression over frame relay, according to industry standard (Frame Relay IP Header Compression Agreement FRF.20), requires a duplex link and negotiations between the compressor router and the decompressor router. In Constellation, each forward to and return link from the CEV in space is treated independently as a simplex link. Without negotiation, the COTS routers are prevented from entering into the IP header compression mode, and no IP header compression would be performed. An algorithm is proposed to enable IP header compression in COTS routers on a simplex link with no negotiation or with a one-way messaging. In doing so, COTS routers can enter IP header compression mode without the need to handshake through a bidirectional link as required by FRF.20. This technique would spoof the routers locally and thereby allow the routers to enter into IP header compression mode without having the negotiations between routers actually occur. The spoofing function is conducted by a frame relay adapter (also COTS) with the capability to generate control messages according to the FRF.20 descriptions. Therefore, negotiation is actually performed between the FRF.20 adapter and the connecting COTS router locally and never occurs over the space link. Through understanding of the handshaking protocol described by FRF.20, the necessary FRF.20 negotiations messages can be generated to control the connecting router, not only to turn on IP header compression but also to adjust the compression parameters. The FRF.20 negotiation (or control) message is composed in the FRF.20 adapter by interpreting the incoming router request message. Many of the fields are simply transcribed from request to response while the control field indicating response and type are modified.

Nguyen, Sam P.↗

Compressing data for storage in cache memories in a hierarchy of cache memories

An electronic device includes at least one compression-decompression functional block and a hierarchy of cache memories with a first cache memory and a second cache memory. The at least one compression-decompression functional block receives data in an uncompressed state, compresses the data using one of a first compression or a second compression, and, after compressing the data, provides the data to the first cache memory for storage therein. When the data is retrieved from the first cache memory to be stored in the second cache memory, when the data is compressed using the first compression, the compression-decompression functional block decompresses the data to reverse effects of the first compression on the data, thereby restoring the data to the uncompressed state and provides the data compressed using the second compression or in the uncompressed state to the second cache memory for storage therein.

97 MATHEMATICS AND COMPUTING↗

MHD simulation on magnetic compression of field reversed configurations with NIMROD

Abstract Magnetic compression has long been proposed a promising method for plasma heating in a field reversed configuration (FRC). However, it remains a challenge to fully understand the physical mechanisms underlying the compression process, due to its highly dynamic nature beyond the one-dimensional (1D) adiabatic theory model (Spencer et al 1983 Phys. Fluids 26 1564). In this work, magnetohydrodynamics simulations on the magnetic compression of FRCs using the NIMROD code (Sovinec et al 2004 J. Comput. Phys. 195 355) and their comparisons with the 1D theory have been performed. The effects of the assumptions of the theory on the compression process have been explored, and the detailed profiles of the FRC during compression have been investigated. The pressure evolution agrees with the theoretical prediction under various initial conditions. The axial contraction of the FRC can be affected by the initial density profile and the ramping rate of the compression magnetic field, but the theoretical predictions on the FRC’s length in general and the relation r s = 2 r o in particular hold approximately well during the whole compression process, where r s is the major radius of FRC separatrix and r o is that of the magnetic axis. The evolutions of the density and temperature can be affected significantly by the initial equilibrium profile and the ramping rate of the compression magnetic field. During the compression, the major radius of the FRC is another parameter that is susceptible to the ramping rate of the compression field. Basically, for the same magnetic compression ratio, the peak density is higher and the FRC’s radius r s is smaller than the theoretical predictions.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Temperature-induced densification in compressed basaltic glass revealed by in-situ ultrasonic measurements

Abstract Acoustic velocities of a model basalt glass (64 mol% CaMgSi2O6 + 36 mol% CaAl2Si2O8) were measured along different pressure-temperature (P-T) paths. One set of experiments involved isothermal compression-decompression cycles, performed at temperatures of 300, 641, 823, and 1006 K and pressures up to 12.2 GPa. The other set of experiments involved constant-load heating-cooling cycles at temperatures up to 823 K and pressures up to 7.5 GPa. Both sets of experiments were performed in a multi-anvil apparatus using a synchrotron-based ultrasonic technique. Our results show that the glass compressed isothermally at 300 K (cold-compression) displays anomalously decreasing compressional (VP) and shear (VS) wave velocities with increasing pressure until ~8 GPa. Beyond 8 GPa, both VP and VS start to increase sharply with pressure and irreversible densification of the glass occurred, producing large hysteresis loops of velocities upon decompression. However, for the glass compressed isothermally at increasingly higher temperatures (hot-compression), the velocity minima gradually shift to lower pressures. At temperature close to the glass transition temperature Tg, the velocity minima disappear completely, displaying a monotonic increase of velocities during compression and higher VP and VS during decompression. In addition, constant-load heating-cooling experiments show that velocities generally decrease slightly with increasing temperature, but start to increase once heated above a threshold temperature (~650 K). During cooling the velocities increase almost linearly with decreasing temperature, resulting in higher velocities (~1.5–2.5% higher) when returned to 300 K. This implies that a temperature-induced densification may have occurred in the glass at high pressures. Raman spectra on recovered samples show that the hot-compressed and high-P heated glasses contain distinctly densified and depolymerized structural signatures compared to the initial glass and the cold-compressed glass below the velocity transition pressure PT (~8 GPa). Such densification may be attributed to the breaking of bridging oxygen bonds and compaction in the intermediate-range structure. Our results demonstrate that temperature can facilitate glass densification at high pressures and point out the importance of P-T history in understanding the elastic properties of silicate glasses. Comparison with melt velocity suggests that hot-compressed glasses may better resemble the pressure dependence of velocity of silicate melts than cold-compressed glasses, but still show significantly higher velocities than melts. If the abnormal acoustic behaviors of cold-compressed glasses were used to constrain melt fractions in the mantle low-velocity regions, the melt fractions needed to explain a given velocity reduction would be significantly underestimated at high pressures.

Geochemistry & Geophysics↗

Some Results Relevant to Statistical Closures for Compressible Turbulence

For weakly compressible turbulent fluctuations there exists a small parameter, the square of the fluctuating Mach number, that allows an investigation using a perturbative treatment. The consequences of such a perturbative analysis in three different subject areas are described: 1) initial conditions in direct numerical simulations, 2) an explanation for the oscillations seen in the compressible pressure in the direct numerical simulations of homogeneous shear, and 3) for turbulence closures accounting for the compressibility of velocity fluctuations. Initial conditions consistent with small turbulent Mach number asymptotics are constructed. The importance of consistent initial conditions in the direct numerical simulation of compressible turbulence is dramatically illustrated: spurious oscillations associated with inconsistent initial conditions are avoided, and the fluctuating dilatational field is some two orders of magnitude smaller for a compressible isotropic turbulence. For the isotropic decay it is shown that the choice of initial conditions can change the scaling law for the compressible dissipation. A two-time expansion of the Navier-Stokes equations is used to distinguish compressible acoustic and compressible advective modes. A simple conceptual model for weakly compressible turbulence - a forced linear oscillator is described. It is shown that the evolution equations for the compressible portions of turbulence can be understood as a forced wave equation with refraction. Acoustic modes of the flow can be amplified by refraction and are able to manifest themselves in large fluctuations of the compressible pressure.

Ristorcelli, J. R.↗

Survey of Header Compression Techniques

This report provides a summary of several different header compression techniques. The different techniques included are: (1) Van Jacobson's header compression (RFC 1144); (2) SCPS (Space Communications Protocol Standards) header compression (SCPS-TP, SCPS-NP); (3) Robust header compression (ROHC); and (4) The header compression techniques in RFC2507 and RFC2508. The methodology for compression and error correction for these schemes are described in the remainder of this document. All of the header compression schemes support compression over simplex links, provided that the end receiver has some means of sending data back to the sender. However, if that return path does not exist, then neither Van Jacobson's nor SCPS can be used, since both rely on TCP (Transmission Control Protocol). In addition, under link conditions of low delay and low error, all of the schemes perform as expected. However, based on the methodology of the schemes, each scheme is likely to behave differently as conditions degrade. Van Jacobson's header compression relies heavily on the TCP retransmission timer and would suffer an increase in loss propagation should the link possess a high delay and/or bit error rate (BER). The SCPS header compression scheme protects against high delay environments by avoiding delta encoding between packets. Thus, loss propagation is avoided. However, SCPS is still affected by an increased BER (bit-error-rate) since the lack of delta encoding results in larger header sizes. Next, the schemes found in RFC2507 and RFC2508 perform well for non-TCP connections in poor conditions. RFC2507 performance with TCP connections is improved by various techniques over Van Jacobson's, but still suffers a performance hit with poor link properties. Also, RFC2507 offers the ability to send TCP data without delta encoding, similar to what SCPS offers. ROHC is similar to the previous two schemes, but adds additional CRCs (cyclic redundancy check) into headers and improves compression schemes which provide better tolerances in conditions with a high BER.

Ishac, Joseph↗

Efficacy of compression of different capacitance beds in the amelioration of orthostatic hypotension

Orthostatic hypotension (OH) is the most disabling and serious manifestation of adrenergic failure, occurring in the autonomic neuropathies, pure autonomic failure (PAF) and multiple system atrophy (MSA). No specific treatment is currently available for most etiologies of OH. A reduction in venous capacity, secondary to some physical counter maneuvers (e.g., squatting or leg crossing), or the use of compressive garments, can ameliorate OH. However, there is little information on the differential efficacy, or the mechanisms of improvement, engendered by compression of specific capacitance beds. We therefore evaluated the efficacy of compression of specific compartments (calves, thighs, low abdomen, calves and thighs, and all compartments combined), using a modified antigravity suit, on the end-points of orthostatic blood pressure, and symptoms of orthostatic intolerance. Fourteen patients (PAF, n = 9; MSA, n = 3; diabetic autonomic neuropathy, n = 2; five males and nine females) with clinical OH were studied. The mean age was 62 years (range 31-78). The mean +/- SEM orthostatic systolic blood pressure when all compartments were compressed was 115.9 +/- 7.4 mmHg, significantly improved (p < 0.001) over the head-up tilt value without compression of 89.6 +/- 7.0 mmHg. The abdomen was the only single compartment whose compression significantly reduced OH (p < 0.005). There was a significant increase of peripheral resistance index (PRI) with compression of abdomen (p < 0.001) or all compartments (p < 0.001); end-diastolic index and cardiac index did not change. We conclude that denervation increases vascular capacity, and that venous compression improves OH by reducing this capacity and increasing PRI. Compression of all compartments is the most efficacious, followed by abdominal compression, whereas leg compression alone was less effective, presumably reflecting the large capacity of the abdomen relative to the legs.

Non-NASA Center↗

Context Modeler for Wavelet Compression of Spectral Hyperspectral Images

A context-modeling sub-algorithm has been developed as part of an algorithm that effects three-dimensional (3D) wavelet-based compression of hyperspectral image data. The context-modeling subalgorithm, hereafter denoted the context modeler, provides estimates of probability distributions of wavelet-transformed data being encoded. These estimates are utilized by an entropy coding subalgorithm that is another major component of the compression algorithm. The estimates make it possible to compress the image data more effectively than would otherwise be possible. The following background discussion is prerequisite to a meaningful summary of the context modeler. This discussion is presented relative to ICER-3D, which is the name attached to a particular compression algorithm and the software that implements it. The ICER-3D software is summarized briefly in the preceding article, ICER-3D Hyperspectral Image Compression Software (NPO-43238). Some aspects of this algorithm were previously described, in a slightly more general context than the ICER-3D software, in "Improving 3D Wavelet-Based Compression of Hyperspectral Images" (NPO-41381), NASA Tech Briefs, Vol. 33, No. 3 (March 2009), page 7a. In turn, ICER-3D is a product of generalization of ICER, another previously reported algorithm and computer program that can perform both lossless and lossy wavelet-based compression and decompression of gray-scale-image data. In ICER-3D, hyperspectral image data are decomposed using a 3D discrete wavelet transform (DWT). Following wavelet decomposition, mean values are subtracted from spatial planes of spatially low-pass subbands prior to encoding. The resulting data are converted to sign-magnitude form and compressed. In ICER-3D, compression is progressive, in that compressed information is ordered so that as more of the compressed data stream is received, successive reconstructions of the hyperspectral image data are of successively higher overall fidelity.

Kiely, Aaron↗

Localized material compression to correct distortion in wire arc additive manufacturing

Wire Arc Additive Manufacturing (WAAM) is an advanced manufacturing technology which utilizes welding systems to generate three dimensional geometries in a layer-by-layer fashion. Distortion or warping of a print substrate and WAAM components due to thermally induced residual stresses is an ongoing challenge limiting the widespread adoption of WAAM technologies for producing components. In this manuscript, a novel approach is described to address thermal distortion in deposited components by applying lateral compressions along the length of the deposited material. To demonstrate this method, a series of single-track walls were printed and compressed at evenly spaced intervals using a modified hydraulic cutter tool. The jaws of the tool were modified to compress material rather than to shear it. A mathematical model was developed to relate the curvature of the deposited material to the volume of compression required to eliminate this distortion. Validation of this model was performed using 3D scan data to compare the change in wall curvature induced by compression to the volume of the applied compressions. Substrate deflection was also compared against a control wall, and implementation of wall compression reduced maximum deflections by 93% across a series of four depositions and subsequent compressions. Wall cross sections were also analyzed to determine the impact of compression on material hardness and grain structure. The results demonstrate that successively placed lateral compressions can effectively control and potentially eliminate bending distortion in printed parts. This methodology can be further developed to form a robust model for correction of thermally-induced distortion in WAAM components.

Additive manufacturing↗

Scalable Hybrid Learning Techniques for Scientific Data Compression

Data compression is becoming critical for storing scientific data because many scientific applications need to store large amounts of data and post process this data for scientific discovery. Unlike image and video compression algorithms that limit errors to primary data (PD), scientists require compression techniques that accurately preserve derived quantities of interest (QoIs). Here, this article presents a physics-informed compression technique implemented as an end-to-end, scalable, GPU-based pipeline for data compression that addresses this requirement. Our hybrid compression technique combines machine learning techniques and standard compression methods. Specifically, we combine an autoencoder, an error-bounded lossy compressor to provide guarantees on raw data error, and a constraint satisfaction post-processing step to preserve the QoIs within a minimal error (generally less than floating point error). The effectiveness of the data compression pipeline is demonstrated by compressing nuclear fusion simulation data generated by a large-scale fusion code, XGC, which produces hundreds of terabytes of data in a single day. Our approach works within the ADIOS framework and results in compression by a factor of more than 150 while requiring only a few percent of the computational resources necessary for generating the data, making the overall approach highly effective for practical scenarios.

ITER↗

Neural-Based Compression Scheme for Solar Image Data

Studying the solar system and especially the Sun relies on the data gathered daily from space missions. These missions are data-intensive and compressing this data to make them efficiently transferable to the ground station is a twofold decision to make. Stronger compression methods, by distorting the data, can increase data throughput at the cost of accuracy which could affect scientific analysis of the data. On the other hand, preserving subtle details in the compressed data requires a high amount of data to be transferred, reducing the desired gains from compression. In this work, we propose a neural network-based lossy compression method to be used in NASA’s data-intensive imagery missions. We chose NASA’s Solar Dynamics Observatory (SDO) mission which transmits 1.4 terabytes of data each day as a proof of concept for the proposed algorithm. In this work, we propose an adversarially trained neural network, equipped with local and non-local attention modules to capture both the local and global structure of the image resulting in a better trade-off in rate-distortion (RD) compared to conventional hand-engineered codecs. The RD variational autoencoder used in this work is jointly trained with a channel-dependent entropy model as a shared prior between the analysis and synthesis transforms to make the entropy coding of the latent code more effective. We also studied how optimizing perceptual losses could help our neural compressor to preserve high-frequency details of the data in the reconstructed compressed image. Our neural image compression algorithm outperforms currently-in-use and state-of-the-art codecs such as JPEG and JPEG-2000 in terms of the RD performance when compressing extreme-ultraviolet (EUV) data. As a proof of concept for use of this algorithm in SDO data analysis, we have performed coronal hole (CH) detection using our compressed images, and generated consistent segmentations, even at a compression rate of ∼ 0.1 bits per pixel (compared to 8 bits per pixel on the original data) using EUV data from SDO.

Image coding↗

The Emerging Issue-3 Revision of the Ccsds-123.0-B Low-Complexity Lossless and Near-Lossless Multispectral and Hyperspectral Image Compression Standard

The CCSDS-123.0-B Low-Complexity Lossless and Near-Lossless Multispectral & Hyperspectral Image Compression standard provides state-of-the-art compression for imaging spectrometer data. Issue 1 of this standard, published in 2012, provides only lossless compression capability. In 2019, Issue 2 was published, adding new features and in particular augmenting the compressor to provide also near-lossless compression, i.e., compression with a user-specified error limit in each spectral band. This presentation will describe the Issue 3 revision currently under development by the CCSDS Data Compression working group. This revision will provide region-of-interest compression capability. That is, a user-provided spatial classification map can be used to specify different data fidelity parameters in different image regions. This would, for example, allow an onboard classification algorithm to identify the most important regions of an image, or regions obscured by clouds, and adjust compression fidelity accordingly in each region to maximize value of imaging spectrometer data returned over constrained space communication links. The revision also aims to add an option that is unrelated to region-of-interest compression but can improve compression effectiveness in some cases.

CCSDS Hyperspectral↗

Model of ramp compression of diamond from ab initio simulations

Ramp compression experiments characterize high-pressure states of matter at temperatures well below those present in shock compression. However, because temperature is typically not directly measured during ramp compression, it is uncertain how much heating occurs under these shock-free conditions. Here, we performed a series of ab initio simulations on carbon in order to match the density-stress measurements of Smith et al. [Smith et al., Nature (London) 511, 330 (2014)]. We considered isotropically as well as uniaxially compressed solid carbon in the diamond and BC8 phases, with and without defects, as well as liquid carbon. Our idealized model ascribes heating during ramp compression to an initially uniaxially compressed cell transforming isochorically into an isotropically (hydrostatic equivalent) compressed state having lower internal energy, hence higher temperature so as to conserve energy. Multiple such heating events can occur during a single ramp experiment, leading to higher temperatures than with isentropic compression. Comparison with experiments shows that heating alone does not explain the equation of state measurements on diamond, instead implying that a significant uniaxial stress component remains present at high compression. The temperature predictions of our ramp compression model remain to be verified with future laboratory measurements.

58 GEOSCIENCES↗

An Algorithmic and Software Pipeline for Very Large Scale Scientific Data Compression with Error Guarantees

Efficient data compression is becoming increasingly critical for storing scientific data because many scientific applications produce vast amounts of data. This paper presents an end-to-end algorithmic and software pipeline for data compression that guarantees both error bounds on primary data (PD) and derived data, known as Quantities of Interest (QoI).We demonstrate the effectiveness of the pipeline by compressing fusion data generated by a large-scale fusion code, XGC, which produces tens of petabytes of data in a single day. We demonstrate that the compression is conducted by setting aside computational resources known as staging nodes, and does not impact the simulation performance. For efficient parallel I/O, the pipeline uses ADIOS2, which many codes such as XGC already use for their parallel I/O. We show that our approach can compress the data by two orders of magnitude while guaranteeing high accuracy on both the PD and the QoIs. Further, the amount of resources required by compression is a few percent of the resources required by simulation while ensuring that the compression time for each stage is less than the corresponding simulation time.This pipeline consists of three main steps. The first step decomposes the data using domain decomposition into small subdomains. Each subdomain is then compressed independently to achieve a high level of parallelism. The second step uses existing techniques that guarantee error bounds on the primary data for each subdomain. The third step uses a post-processing optimization technique based on Lagrange multipliers to reduce the QoI errors for data corresponding to each subdomain. The Lagrange multipliers generated can be further quantized or truncated to increase the compression level. All of the above characteristics of our approach make it highly practical to apply on-the-fly compression while guaranteeing errors on QoIs that are critical to the scientists.

Banerjee, Tania↗

The effect of compression on individual pressure vessel nickel/hydrogen components

Compression tests were performed on representative Individual Pressure Vessel (IPV) Nickel/Hydrogen cell components in an effort to better understand the effects of force on component compression and the interactions of components under compression. It appears that the separator is the most easily compressed of all of the stack components. It will typically partially compress before any of the other components begin to compress. The compression characteristics of the cell components in assembly differed considerably from what would be predicted based on individual compression characteristics. Component interactions played a significant role in the stack response to compression. The results of the compression tests were factored into the design and selection of Belleville washers added to the cell stack to accommodate nickel electrode expansion while keeping the pressure on the stack within a reasonable range of the original preset.

Manzo, Michelle A.↗