Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35

Compressive Sensing Based Data Acquisition Architecture for Transient Stellar Events in Crowded Star Fields

Compressive sensing is a mathematical technique for simultaneous data acquisition and compression. In this work, we show a CS based architecture for acquiring and reconstructing transient stellar events. This architecture recon-structs a differenced image itself, eliminating the need for any sparse domain transforms, otherwise required for traditional CS reconstruction. The resulting reconstructed differenced image is of critical importance as the information required for generating a time-series photometric light curve is obtained only from a differenced image. Hence, reconstructing a crowded star spatial image, followed by differencing is wasteful. This architecture eliminates the need to 1.) transform an image to a sparse domain, 2.) Reconstruct a dense field, and then apply differencing on the image to obtain the image of critical value. We study the case of microlensing to depict a star source experiencing magnification in time. Our results show that this architecture is able to reconstruct star source magnitudes with magnification factors greater than 1 for clean images with error less than 2% using only 10% of the Nyquist rate samples.

Asmita Korde-patel↗

Experiences with making diffraction image data available: what metadata do we need to archive?

A local raw ‘diffraction data images’ archive was made available and some data sets were retrieved and reprocessed, which led to analysis of the anomalous difference densities of two partially occupied Cl atoms in cisplatin as well as a re-evaluation of the resolution cutoff in these diffraction data. General questions on storing raw data are discussed. It is also demonstrated that often one needs unambiguous prior knowledge to read the (binary) detector format and the setup of goniometer geometries. Recently, the IUCr (International Union of Crystallography) initiated the formation of a Diffraction Data Deposition Working Group with the aim of developing standards for the representation of raw diffraction data associated with the publication of structural papers. Archiving of raw data serves several goals: to improve the record of science, to verify the reproducibility and to allow detailed checks of scientific data, safeguarding against fraud and to allow reanalysis with future improved techniques. A means of studying this issue is to submit exemplar publications with associated raw data and metadata. In a recent study of the binding of cisplatin and carboplatin to histidine in lysozyme crystals under several conditions, the possible effects of the equipment and X-ray diffraction data-processing software on the occupancies and B factors of the bound Pt compounds were compared. Initially, 35.3 GB of data were transferred from Manchester to Utrecht to be processed with EVAL. A detailed description and discussion of the availability of metadata was published in a paper that was linked to a local raw data archive at Utrecht University and also mirrored at the TARDIS raw diffraction data archive in Australia. By making these raw diffraction data sets available with the article, it is possible for the diffraction community to make their own evaluation. This led to one of the authors of XDS (K. Diederichs) to re-integrate the data from crystals that supposedly solely contained bound carboplatin, resulting in the analysis of partially occupied chlorine anomalous electron densities near the Pt-binding sites and the use of several criteria to more carefully assess the diffraction resolution limit. General arguments for archiving raw data, the possibilities of doing so and the requirement of resources are discussed. The problems associated with a partially unknown experimental setup, which preferably should be available as metadata, is discussed. Current thoughts on data compression are summarized, which could be a solution especially for pixel-device data sets with fine slicing that may otherwise present an unmanageable amount of data.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗

Evaluation of the Radiometric Quality of the TM Data Using Clustering, Linear Transformations and Multispectral Distance Measures

The radiometric quality of LANDSAT 4 TM data for the classification and identification of Earth surface features was evaluated. Techniques employed in the evaluation included clustering, data compression (linear transformations), multispectral distance measures, and hierarchical classification methods. TM and MSS data for the Chicago, Illinois test site were studied. In order to determine the radiometric quality of the TM thermal data for temperature mapping of surface water, a test site was selected within the area covered by the TM scene (Scene ID: 40101-16025) gathered over Illinois. This site was chosen because it includes a surface water body with a large range of temperatures, i.e., a cooling pond for the Dresden nuclear power plant and the junction of two rivers.

Bartolucci, L. A.↗

Conference on Adaptive Telemetry

Adaptive telemetry conference on adaptive data control and processing, prediction, data compression, encoding, decoding, and modulation.

TELEMETRY↗

Cinema:Snap: Real-time tools for analysis of dynamic diamond anvil cell experiment data

We report we developed tools and a workflow for real-time analysis of data from dynamic diamond anvil cell experiments performed at user light sources. These tools allow users to determine the phases of matter observed during the compression of materials in order to make decisions during an experiment to improve the quality of experimental results and maximize the use of scarce experimental facility time. The tools fill a gap in dynamic compression data analysis tools that are real-time, are flexible to the needs of high-pressure scientists, connect to automated processing of results, can be easily incorporated into workflows with existing tools and data formats, and support remote experimental data analysis workflows. Specific analytics developed include novel automated two-peak analysis for overlapping peaks and multiple phases, coordinated views of pressure and temperature values, full-compression contour plots, and configurable views of integrated x-ray diffraction. We present an experimental use case to show how the tools produce real-time analytics that help the scientists revise parameters for the next compression

47 OTHER INSTRUMENTATION↗

Machine Learning Techniques for Data Reduction of Climate Applications

Scientists conduct large-scale simulations to compute derived quantities-of-interest (QoI) from primary data. Often, QoI are linked to specific features, regions, or time intervals, such that data can be adaptively reduced without compromising the integrity of QoI. For many spatiotemporal applications, these QoI are binary in nature and represent presence or absence of a physical phenomenon. We present a pipelined compression approach that first uses neural-network-based techniques to derive regions where QoI are highly likely to be present. Then, we employ a Guaranteed Autoencoder (GAE) to compress data with differential error bounds. GAE uses QoI information to apply low-error compression to only these regions. This results in overall high compression ratios while still achieving downstream goals of simulation or data collections. Experimental results are presented for climate data generated from the E3SM Simulation model for downstream quantities such as tropical cyclone and atmospheric river detection and tracking. These results show that our approach is superior to comparable methods in the literature.

Li, Xiao [University of Florida]↗

Rocketdyne automated dynamics data analysis and management system

An automated dynamics data analysis and management systems implemented on a DEC VAX minicomputer cluster is described. Multichannel acquisition, Fast Fourier Transformation analysis, and an online database have significantly improved the analysis of wideband transducer responses from Space Shuttle Main Engine testing. Leakage error correction to recover sinusoid amplitudes and correct for frequency slewing is described. The phase errors caused by FM recorder/playback head misalignment are automatically measured and used to correct the data. Data compression methods are described and compared. The system hardware is described. Applications using the data base are introduced, including software for power spectral density, instantaneous time history, amplitude histogram, fatigue analysis, and rotordynamics expert system analysis.

Tarn, Robert B.↗

Design of joint source/channel coders

The need to transmit large amounts of data over a band limited channel has led to the development of various data compression schemes. Many of these schemes function by attempting to remove redundancy from the data stream. An unwanted side effect of this approach is to make the information transfer process more vulnerable to channel noise. Efforts at protecting against errors involve the reinsertion of redundancy and an increase in bandwidth requirements. The papers presented within this document attempt to deal with these problems from a number of different approaches.

Source record↗

Compressing subbanded image data with Lempel-Ziv-based coders

A method of improving the compression of image data using Lempel-Ziv-based coding is presented. Image data is first processed with a simple transform, such as the Walsh Hadamard Transform, to produce subbands. The subbanded data can be rounded to eight bits or it can be quantized for higher compression at the cost of some reduction in the quality of the reconstructed image. The data is then run-length coded to take advantage of the large runs of zeros produced by quantization. Compression results are presented and contrasted with a subband compression method using quantization followed by run-length coding and Huffman coding. The Lempel-Ziv-based coding in conjunction with run-length coding produces the best compression results at the same reconstruction quality (compared with the Huffman-based coding) on the image data used.

Glover, Daniel↗

Performance Optimization for Relative-Error-Bounded Lossy Compression on Scientific Data

Scientific simulations in high-performance computing (HPC) environments generate vast volume of data, which may cause a severe I/O bottleneck at runtime and a huge burden on storage space for postanalysis. Unlike traditional data reduction schemes such as deduplication or lossless compression, not only can error-controlled lossy compression significantly reduce the data size but it also holds the promise to satisfy user demand on error control. Pointwise relative error bounds (i.e., compression errors depends on the data values) are widely used by many scientific applications with lossy compression since error control can adapt to the error bound in the dataset automatically. Pointwise relative-error-bounded compression is complicated and time consuming. In this article, we develop efficient precomputation-based mechanisms based on the SZ lossy compression framework. Our mechanisms can avoid costly logarithmic transformation and identify quantization factor values via a fast table lookup, greatly accelerating the relative-error-bounded compression with excellent compression ratios. In addition, we reduce traversing operations for Huffman decoding, significantly accelerating the decompression process in SZ. Experiments with eight well-known real-world scientific simulation datasets show that our solution can improve the compression and decompression rates (i.e., the speed) by about 40 and 80 p, respectively, in most of cases, making our designed lossy compression strategy the best-in-class solution in most cases.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗

Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound

An effective data compressor is becoming increasingly critical to today's scientific research, and many lossy compressors are developed in the context of absolute error bounds. Based on physical/chemical definitions of simulation fields or multiresolution demand, however, many scientific applications need to compress the data with a pointwise relative error bound (i.e., the smaller the data value, the smaller the compression error to tolerate). To this end, we propose two optimized lossy compression strategies under a state-of-the-art three-staged compression framework (prediction + quantization + entropy-encoding). The first strategy (called block-based strategy) splits the data set into many small blocks and computes an absolute error bound for each block, so it is particularly suitable for the data with relatively high consecutiveness in space. The second strategy (called multi-threshold-based strategy) splits the whole value range into multiple groups with exponentially increasing thresholds and performs the compression in each group separately, which is particularly suitable for the data with a relatively large value range and spiky value changes. We implement the two strategies rigorously and evaluate them comprehensively by using two scientific applications which both require lossy compression with point-wise relative error bound. Experiments show that the two strategies exhibit the best compression qualities on different types of data sets respectively. In conclusion, the compression ratio of our lossy compressor is higher than that of other state-of-the-art compressors by 17.2-618 percent on the climate simulation data and 30-210 percent on the N-body simulation data, with the same relative error bound and without degradation of the overall visualization effect of the entire data.

97 MATHEMATICS AND COMPUTING↗

Shock compression of poly(methyl methacrylate) PMMA in the 1000 GPa regime: Z machine experiments

Hydrocarbon polymers are used in a wide variety of practical applications. In the field of dynamic compression at extreme pressures, these polymers are used at several high energy density (HED) experimental facilities. One of the most common polymers is poly(methyl methacrylate) or PMMA, also called Plexiglass® or Lucite®. Here, we present high-fidelity, hundreds of GPa range experimental shock compression data measured on Sandia’s Z machine. We extend the principal shock Hugoniot for PMMA to more than threefold compression up to 650 GPa and re-shock Hugoniot states up to 1020 GPa in an off-Hugoniot regime, where experimental data are even sparser. These data can be used to put additional constraints on tabular equation of state (EOS) models. The present results provide clear evidence for the need to re-examine the existing tabular EOS models for PMMA above ~120 GPa as well as perhaps revisit EOSs of similar hydrocarbon polymers commonly used in HED experiments investigating dynamic compression, hydrodynamics, or inertial confinement fusion.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Kinetic and temperature sensitivity properties of soil exoenzymes through the soil profile down to one-meter depth at a temperate coniferous forest (Blodgett, CA)

This dataset contains data on kinetic and temperature sensitivity parameters of the exoenzymes β-glucosidase (BG), leucine/leucyl aminopeptidase (LAP) and acid phosphatase (AP) across a soil profile down to 90 cm depth at Blodgett forest, CA, USA. These data were generated to determine if kinetic and thermal properties of microbial exoenzymes involved in organic matter decomposition varied with soil depth, following variation in soil properties and microbial communities, as part of the study: Alves et al. (2021). Kinetic Properties of Microbial Exoenzymes Vary with Soil Depth but Have Similar Temperature Sensitivities Through the Soil Profile. Frontiers in Microbiology 12:3618. https://doi.org/10.3389/fmicb.2021.735282. This research was performed within the framework of the TES Belowground Biogeochemistry SFA project, in particular association with a long-term field warming experiment of the whole soil profile at Blodgett forest. Samples for this work were collected from locations representative of the field experimental plots. Potential enzyme activity rates were measured using laboratory fluorometric assays with soils collected at 0-10, 10-20, 30-40, 50-60, 60-70 and 80-90 cm deep in biological triplicates (i.e., three soil cores collected at different representative locations). Assays with each soil were conducted over a gradient of eight substrate concentrations per enzyme, and incubated at 4, 10, 16, 25, 35 or 50°C. Enzyme Michaelis-Menten kinetics were modeled over the eight substrate concentrations at each temperature, and the temperature sensitivity of the kinetic parameters was modeled over the six temperatures using linear Arrhenius/Q10 and non-linear Macromolecular Rate Theory (MMRT) models. The dataset includes the fully processed enzyme activity rate data used to model Michaelis-Menten kinetics, calculated kinetic and temperature sensitivity parameters, and basic soil and microbial biomass chemistry for each sample. All data is provided for each individual biological replicate, and kinetic and temperature sensitivity parameters are also provided as means of the biological replicates. The dataset also includes all raw measurement data and code used to parse, combine and perform the analyses described by Alves et al. (2021). For file descriptions, see the file-level metadata files: “enzymes_dataProcessed_flmd.xlsx” (processed data); “enzymes_dataRaw_flmd.xlsx” (compressed raw data and metadata); and “enzymes_code_flmd.xlsx” (compressed code). The experimental design, list of parameters measured, soil and microbial biomass chemistry data, and means of biological replicates for kinetic and temperature sensitivity parameters are also provided as human-readable tables in file “enzymeTraits_design_results_tables.xlsx”.

54 ENVIRONMENTAL SCIENCES↗

Estimating the size of Huffman code preambles

Data compression via block-adaptive Huffman coding is considered. The compressor consecutively processes blocks of N data symbols, estimates source statistics by computing the relative frequencies of each source symbol in the block, and then synthesizes a Huffman code based on these estimates. In order to let the decompressor know which Huffman code is being used, the compressor must begin the transmission of each compressed block with a short preamble or header file. This file is an encoding of the list n = (n(sub 1), n(sub 2)....,n(sub m)), where n(sub i) is the length of the Hufffman codeword associated with the ith source symbol. A simple method of doing this encoding is to individually encode each n(sub i) into a fixed-length binary word of length log(sub 2)l, where l is an a priori upper bound on the codeword length. This method produces a maximum preamble length of mlog(sub 2)l bits. The object is to show that, in most cases, no substantially shorter header of any kind is possible.

Mceliece, R. J.↗

Extended dynamic mode decomposition for model reduction in fluid dynamics simulations

High computational cost and storage/memory requirements of fluid dynamics simulations constrain their usefulness as a predictive tool. Reduced-order models (ROMs) provide a viable solution to this challenge by extracting the key underlying dynamics of a complex system directly from data. We investigate the efficacy and robustness of an extended dynamic mode decomposition (xDMD) algorithm in constructing ROMs of three-dimensional cardiovascular computations. Focusing on the ROMs' accuracy in representation and interpolation, we relate these metrics to the truncation rank of singular value decomposition, which underpins xDMD and other approaches to ROM construction. Our key innovation is to relate the truncation rank to the singular values of the original flow problem. This result establishes a priori guidelines for the xDMD deployment and its likely success as a means of data compression and reconstruction of the system's dynamics from dominant spatiotemporal structures present in the data.

Mechanics↗

All Optical Neural Networks for Low Power Edge Computing

We developed a simplistic physics-based model of an all-optical neural network that mimics the encoder part of an autoencoder neural network for image compression. Our approach relies on the generation of a MATLAB-based model for both data compression and decompression and utilizes MATLAB's built-in autoencoder networks in combination with simple propagation of optical fields between layers constituting phase elements via Fourier transform. We optimize the phase elements using the particle swarm optimization technique and using our model, we demonstrate a compression ratio of 25% for 2828-pixel input images containing numeric digits from 0 to 9.

97 MATHEMATICS AND COMPUTING↗