Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,063 records · Page 59

Detection And Avoidance Of Obstacles By Helicopters

Report discusses problems relevant to control subsystems enabling helicopters on nap-of-the-Earth flight paths to detect and avoid obstacles automatically. Indicates similarities between this and obstacle-avoidance problem of industrial mobile robots. Two approaches extend two-dimensional obstacle-avoidance concept to three dimensions. First involves direct search of three-dimensional range-map data for indications of openings between obstacles. Second involves compression of data into two-dimensional map for path search.

Cheng, Victor H. L.↗

Earth resources data processor

The recent development of manned and unmanned space vehicles has brought about an almost unprecedented advance in studies concerned with remotely sensed earth observations. With this advance comes an unprecedented amount of data. The problem arises of how to efficiently analyze and compress unmanageable amounts of data into manageable amounts of useful information. A recently developed computer program is proposed as a partial solution to the above problem. The computer program is designed to determine the ground scene location and distribution of features extracted from remotely sensed earth observation data without human involvement in the data processing or a priori knowledge of ground truth. Human involvement and judgement are reserved for identification of the features presented in the compressed data.

Phillips, M. R.↗

A Formal Messaging Notation for Alaskan Aviation Data

Data exchange is an increasingly important aspect of the National Airspace System. While many data communication channels have become more capable of sending and receiving data at higher throughput rates, there is still a need to use communication channels efficiently with limited throughput. The limitation can be based on technological issues, financial considerations, or both. This paper provides a complete description of several important aviation weather data in Abstract Syntax Notation format. By doing so, data providers can take advantage of Abstract Syntax Notation's ability to encode data in a highly compressed format. When data such as pilot weather reports, surface weather observations, and various weather predictions are compressed in such a manner, it allows for the efficient use of throughput-limited communication channels. This paper provides details on the Abstract Syntax Notation One (ASN.1) implementation for Alaskan aviation data, and demonstrates its use on real-world aviation weather data samples as Alaska has sparse terrestrial data infrastructure and data are often sent via relatively costly satellite channels.

communications↗

Analyzing acoustic emission data to identify cracking modes in cement paste using an artificial neural network

This research is focused on the identification of cracking mechanisms for cement paste using acoustic emission data, recorded from compression and notched four-point bending tests. A procedure is developed for analyzing the data by employing an agglomerative hierarchical clustering method, an artificial neural network, and a ray-tracing source location algorithm. An agglomerative hierarchical clustering method is utilized to cluster the AE data from a compression test using frequency-dependent features. A neural network is trained using the compression test data and applied to the AE data emitted during the four-point bending test. The clustered data from the four-point bending test is localized using a ray-tracing algorithm. Based on the occurrence and locations of the clustered events and signal feature analyses, potential cracking mechanisms are identified and assigned.

36 MATERIALS SCIENCE↗

Applying Compression-Based Metrics to Seismic Data in Support of Global Nuclear Explosion Monitoring

The analysis of seismic data for evidence of possible nuclear explosion testing is a critical global security mission that relies heavily on human expertise to identify and mark seismic signals embedded in background noise. To assist analysts in making these determinations, we adapted two compression distance metrics for use with seismic data. First, we demonstrated that the Normalized Compression Distance (NCD) metric can be adapted for use with waveform data and can identify the arrival times of seismic signals. Then we tested an approximation for the NCD called Sliding Information Distance (SLID), which can be computed much faster than NCD. We assessed the accuracy of the SLID output by comparing it to both the Akaike Information Criterion (AIC) and the judgments of expert seismic analysts. Our results indicate that SLID effectively identifies arrival times and provides analysts with useful information that can aid their analysis process.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗

CODARcode/MGARD

MGARD is a software providing error-controlled lossy compression and data refactoring based on multi-grid theories. It transforms floating-point scientific data into a multilevel representation, followed by quantization and lossless encoding processes, resulting in a self-describing compressed buffer. It supports diverse data topologies, error control norms, and computing architectures.

Chen, Jieyang [University of Oregon]↗

Pass-efficient methods for compression of high-dimensional turbulent flow data

The future of high-performance computing, specifically on future Exascale computers, will presumably see memory capacity and bandwidth fail to keep pace with data generated, for instance, from massively parallel partial differential equation (PDE) systems. Current strategies proposed to address this bottleneck entail the omission of large fractions of data, as well as the incorporation of in situ compression algorithms to avoid overuse of memory. To ensure that post-processing operations are successful, this must be done in a way that a sufficiently accurate representation of the solution is stored. Moreover, in situations where the input/output system becomes a bottleneck in analysis, visualization, etc., or the execution of the PDE solver is expensive, the number of passes made over the data must be minimized. In the interest of addressing this problem, this work focuses on the utility of pass-efficient, parallelizable, low-rank, matrix decomposition methods in compressing high-dimensional simulation data from turbulent flows. Additionally, a particular emphasis is placed on using coarse representation of the data – compatible with the PDE discretization grid – to accelerate the construction of the low-rank factorization. This includes the presentation of a novel single-pass matrix decomposition algorithm for computing the so-called interpolative decomposition. The methods are described extensively and numerical experiments on two turbulent channel flow data are performed. In the first (unladen) channel flow case, compression factors exceeding 400 are achieved while maintaining accuracy with respect to first- and second-order flow statistics. In the particle-laden case, compression factors of 100 are achieved and the compressed data is used to recover particle velocities. These results show that these compression methods can enable efficient computation of various quantities of interest in both the carrier and disperse phases.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

A technique to calibrate polarimetric radar images using only image parameters and trihedral corner reflectors

The technique described uses the radar return from natural targets and at least one trihedral corner reflector to calibrate compressed polarimetric radar data for relative amplitude, relative phase, absolute amplitude, and system crosstalk. The crosstalk method is based on the theoretical result that, for natural targets with azimuthal symmetry, the copolarized and cross-polarized components of the scattering matrix are uncorrelated. This method does not require any external calibration targets to be deployed. Since compressed data are used, it is necessary to model the transmitting and receiving systems as reciprocal. The method is used to estimate the crosstalk parameters of the NASA/JPL aircraft for different types of terrain and for two frequencies. For the C-band systems the crosstalk is less than -20 dB for all ranges in the images. The crosstalk of the L-band system is a function of range, however, and may be as poor as -10 dB in the near range, leading to a noticeable distortion of the polarization signatures.

Van Zyl, Jakob J.↗

Maximizing efficiency of dataset compression for machine learning potentials with information theory

Machine learning interatomic potentials (MLIPs) balance high accuracy and lower costs compared to density functional theory calculations, but their performance often depends on the size and diversity of training datasets. Large datasets improve model accuracy and generalization but are computationally expensive to produce and train on, while smaller datasets risk discarding rare but important atomic environments and compromising MLIP accuracy/reliability. Here, we develop an information-theoretical framework to quantify the efficiency of dataset compression methods and propose an algorithm that maximizes this efficiency. By framing atomistic dataset compression as an instance of the minimum set cover (MSC) problem over atom-centered environments, our method identifies the smallest subset of structures that contains as much information as possible from the original dataset while pruning redundant information. The approach is extensively demonstrated on the GAP-20 and TM23 datasets and validated on 64 varied datasets from the ColabFit repository. Across all cases, MSC consistently retains outliers, preserves dataset diversity, and reproduces the long-tail distributions of forces even at high compression rates, outperforming other subsampling methods. Furthermore, MLIPs trained on MSC-compressed datasets exhibit reduced error for out-of-distribution data even in low-data regimes. We explain these results using an outlier analysis and show that such quantitative conclusions could not be achieved with conventional dimensionality reduction methods. The algorithm is implemented in the open-source QUESTS package and can be used for several tasks in atomistic modeling, from data subsampling, outlier detection, and training improved MLIPs at a lower cost.

36 MATERIALS SCIENCE↗

Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition

Extreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time. The challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-the-art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and ~ 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively.

Liang, Xin↗

Adaptive Quantizer for Burst Synthetic-Aperture Radar

Adaptive quantizer for burst-mode synthetic-aperture radar reduces data rate of return signal. Device, called block floating-point quantizer (BFPQ) basically analog-to-digital converter that covers wide dynamic range and discards appropriate lower order bits. BFPQ is, in effect, digital approximator with automatic gain control. Moves floatingpoint marker binary representation of signal data in accordance with perceived dynamic range. Available step sizes thus limited to multiples of underlying smallest quantization step (represented by lowest order bit). Retains only first K most significant bits of signal; (L,K) BFPQ is one that does K-bit quantization of signal originally quantized to L bits. Quantization error simply difference between actual signal level and its binary approximation. Other potential applications for BFPQ include speech compression and picture data compression.

Joo, T. H.↗

Emerging standards for still image compression: A software implementation and simulation study

The software implementation is described of an emerging standard for the lossy compression of continuous tone still images. This software program can be used to compress planetary images and other 2-D instrument data. It provides a high compression image coding capability that preserves image fidelity at compression rates competitive or superior to most known techniques. This software implementation confirms the usefulness of such data compression and allows its performance to be compared with other schemes used in deep space missions and for data based storage.

Pollara, F.↗

CHIPFoam Modification to Allow for Modelling SWIFT Foams

SWIFT (Silicon-water in Familiar Template) foams are silicone foams that have a continuous porosity without having any obvious spherical or ellipsoidal pore shapes. Like other foamed rubber, SWIFT foams can experience large reversible deformations. Because of their microstructure, SWIFT foams have a stiffering behavior in compression that is very abrupt compared to most other foamed rubbers. Work by Benedikt reported by Miller showed that fitting compressive test data with the CHIPFoam model in its current form is impractical and inaccurate. This report describes an extension made to CHIPFoam to allow the modeling of SWIFT foams. While the extension allows the modeling of SWIFT foams, it also extends the range of porosity of foams that can be modelled with CHIPFoam to include very high porosity (>99%) foamed rubber. The previous model was unstable for porosities greater than 70%. The changes made to CHIPFoam include the introduction of a critical porosity at which the compressible Danielsson strain energy function that couples volumetric and deviatoric mechanical behavior begins to increase with compression. This new feature also required the introduction of a third term in the multiplicative decomposition of the relative volume, J . The organization of this report is described here. The first section, SWIFT Foam Structure and Mechanical Behavior, describes the structure of SWIFT foams as contrasted to other foamed rubbers. It also contrasts the mechanical behavior of SWIFT foams to other more traditional foamed rubbers. The second section, CHIPFoam Modifications, describes the specific modifications done to be able to modify the delayed rapid compressive stiffening seen in SWIFT foams. The third section, Results, demonstrates the use of the modified model to fit compressive loading response data from a SWIFT foam sample. The fourth section, Conclusions and Further Work, puts the work in perspective and presents likely further work.

36 MATERIALS SCIENCE↗

FRaZ: A Generic High-Fidelity Fixed-Ratio Lossy Compression Framework for Scientific Floating-point Data

With ever-increasing volumes of scientific floating-point data being produced by high-performance computing applications, significantly reducing scientific floating-point data size is critical, and error-controlled lossy compressors have been developed for years. None of the existing scientific floating-point lossy data compressors, however, support effective fixed-ratio lossy compression. Yet fixed-ratio lossy compression for scientific floating-point data not only compresses to the requested ratio but also respects a user-specified error bound with higher fidelity. In this paper, we present FRaZ: a generic fixed-ratio lossy compression framework respecting user-specified error constraints. The contribution is twofold. (1) We develop an efficient iterative approach to accurately determine the appropriate error settings for different lossy compressors based on target compression ratios. (2) We perform a thorough performance and accuracy evaluation for our proposed fixed-ratio compression framework with multiple state-of-the-art error-controlled lossy compressors, using several real-world scientific floating-point datasets from different domains. Experiments show that FRaZ effectively identifies the optimum error setting in the entire error setting space of any given lossy compressor. While fixed-ratio lossy compression is slower than fixed-error compression, it provides an important new lossy compression technique for users of very large scientific floating-point datasets.

Underwood, Robert↗

A dictionary learning algorithm for compression and reconstruction of streaming data in preset order

There has been an emerging interest in developing and applying dictionary learning (DL) to process massive datasets in the last decade. Many of these efforts, however, focus on employing DL to compress and extract a set of important features from data, while considering restoring the original data from this set a secondary goal. On the other hand, although several methods are able to process streaming data by updating the dictionary incrementally as new snapshots pass by, most of those algorithms are designed for the setting where the snapshots are randomly drawn from a probability distribution. In this paper, we present a new DL approach to compress and denoise massive dataset in real time, in which the data are streamed through in a preset order (instances are videos and temporal experimental data), so at any time, we can only observe a biased sample set of the whole data. Here, our approach incrementally builds up the dictionary in a relatively simple manner: if the new snapshot is adequately explained by the current dictionary, we perform a sparse coding to find its sparse representation; otherwise, we add the new snapshot to the dictionary, with a Gram-Schmidt process to maintain the orthogonality. To compress and denoise noisy datasets, we apply the denoising to the snapshot directly before sparse coding, which deviates from traditional dictionary learning approach that achieves denoising via sparse coding. Compared to full-batch matrix decomposition methods, where the whole data is kept in memory, and other mini-batch approaches, where unbiased sampling is often assumed, our approach has minimal requirement in data sampling and storage: i) each snapshot is only seen once then discarded, and ii) the snapshots are drawn in a preset order, so can be highly biased. Through experiments on climate simulations and scanning transmission electron microscopy (STEM) data, we demonstrate that the proposed approach performs competitively to those methods in data reconstruction and denoising.

97 MATHEMATICS AND COMPUTING↗

Radio astronomy Explorer B antenna aspect processor

The antenna aspect system used on the Radio Astronomy Explorer B spacecraft is described. This system consists of two facsimile cameras, a data encoder, and a data processor. Emphasis is placed on the discussion of the data processor, which contains a data compressor and a source encoder. With this compression scheme a compression ratio of 8 is achieved on a typical line of camera data. These compressed data are then convolutionally encoded.

Miller, W. H.↗

Processing SAR Images On Board

Synthetic-aperture radar (SAR) processor operates aboard moving radar platform, turning raw signal data into images of scanned terrain. In conventional SAR systems, raw data either transmitted to stations on ground or recorded on magnetic tape by high-density digital recorders for subsequent processing on ground. Advantages include making SAR images available immediately and processed data transmitted or recorded only one-fifth as voluminous as raw data. Onboard image compression reduces volume of data even further.

Liu, Kuang Y.↗

A comparison of three SAR interference filters

Three interference filters are discussed and their ability to remove the interference while leaving target signal data is quantified. The filters are adaptive frequency-domain spike filtering; adaptive frequency-domain interference determination and masking; and interference-target correlation masking. Using Jet Propulsion Laboratory Airborne Synthetic Aperture Radar (JPL AIRSAR) data, the algorithms are applied to three data sets. The resultant compressed Stokes matrix data are compared with the normally processed interference-free image in a quantitative sense. The algorithms are evaluated by measuring the polarization signature, SNR, equivalent number of looks, and impulse response of the processed images.

Chapman, B.↗