Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 883 records · Page 49
All sourcesNASA NTRSDOE OSTI
NASA NTRS1992

Issues in knowledge representation to support maintainability: A case study in scientific data preparation

Scientific data preparation is the process of extracting usable scientific data from raw instrument data. This task involves noise detection (and subsequent noise classification and flagging or removal), extracting data from compressed forms, and construction of derivative or aggregate data (e.g. spectral densities or running averages). A software system called PIPE provides intelligent assistance to users developing scientific data preparation plans using a programming language called Master Plumber. PIPE provides this assistance capability by using a process description to create a dependency model of the scientific data preparation plan. This dependency model can then be used to verify syntactic and semantic constraints on processing steps to perform limited plan validation. PIPE also provides capabilities for using this model to assist in debugging faulty data preparation plans. In this case, the process model is used to focus the developer's attention upon those processing steps and data elements that were used in computing the faulty output values. Finally, the dependency model of a plan can be used to perform plan optimization and runtime estimation. These capabilities allow scientists to spend less time developing data preparation procedures and more time on scientific analysis tasks. Because the scientific data processing modules (called fittings) evolve to match scientists' needs, issues regarding maintainability are of prime importance in PIPE. This paper describes the PIPE system and describes how issues in maintainability affected the knowledge representation used in PIPE to capture knowledge about the behavior of fittings.

Chien, Steve↗
NASA NTRS1993

Intelligent assistance in scientific data preparation

Scientific data preparation is the process of extracting usable scientific data from raw instrument data. This task involves noise detection (and subsequent noise classification and flagging or removal), extracting data from compressed forms, and construction of derivative or aggregate data (e.g. spectral densities or running averages). A software system called PIPE provides intelligent assistance to users developing scientific data preparation plans using a programming language called Master Plumber. PIPE provides this assistance capability by using a process description to create a dependency model of the scientific data preparation plan. This dependency model can then be used to verify syntactic and semantic constraints on processing steps to perform limited plan validation. PIPE also provides capabilities for using this model to assist in debugging faulty data preparation plans. In this case, the process model is used to focus the developer's attention upon those processing steps and data elements that were used in computing the faulty output values. Finally, the dependency model of a plan can be used to perform plan optimization and run time estimation. These capabilities allow scientists to spend less time developing data preparation procedures and more time on scientific analysis tasks.

Chien, Steve↗
NASA NTRS1996

Digital Data Registration and Differencing Compression System

A process for X-ray registration and differencing results in more efficient compression. Differencing of registered modeled subject image with a modeled reference image forms a differenced image for compression with conventional compression algorithms. Obtention of a modeled reference image includes modeling a relatively unrelated standard reference image upon a three-dimensional model, which three-dimensional model is also used to model the subject image for obtaining the modeled subject image. The registration process of the modeled subject image and modeled reference image translationally correlates such modeled images for resulting correlation thereof in spatial and spectral dimensions. Prior to compression, a portion of the image falling outside a designated area of interest may be eliminated, for subsequent replenishment with a standard reference image. The compressed differenced image may be subsequently transmitted and/or stored, for subsequent decompression and addition to a standard reference image so as to form a reconstituted or approximated subject image at either a remote location and/or at a later moment in time. Overall effective compression ratios of 100:1 are possible for thoracic X-ray digital images.

Ransford, Gary A.↗
NASA NTRS1990

Calibration of polarimetric radar images using only image parameters and trihedral corner reflector responses

A technique that uses the radar return from natural targets and at least one trihedral corner reflector to calibrate compressed polarimetric radar data is described. Calibration for relative amplitude and phase, absolute amplitude, and system crosstalk is addressed. The crosstalk calibration method does not require any external calibration targets. Because compressed data are used, the transmitting and receiving systems as must be modeled as reciprocal. Even though the inferred transmit and receive matrices are not simply related to the physical systems, the true Stokes maxtrix for each pixel can be determined accurately. The method is illustrated by estimating the crosstalk parameters of the NASA/JPL aircraft for different types of terrain and for two frequencies. For a C-band system, the crosstalk is less than -20 dB for all ranges in the images. The crosstalk of the L-band system is a function of range and may be as poor as -10 dB in the near range, leading to a noticeable distortion of the polarization signatures.

Van Zyl, Jakob J.↗
NASA NTRS1976

On-board image compression for the RAE lunar mission

The requirements, design, implementation, and flight performance of an on-board image compression system for the lunar orbiting Radio Astronomy Explorer-2 (RAE-2) spacecraft are described. The image to be compressed is a panoramic camera view of the long radio astronomy antenna booms used for gravity-gradient stabilization of the spacecraft. A compression ratio of 32 to 1 is obtained by a combination of scan line skipping and adaptive run-length coding. The compressed imagery data are convolutionally encoded for error protection. This image compression system occupies about 1000 cu cm and consumes 0.4 W.

Miller, W. H.↗
NASA NTRS1978

Simplified data compressor

Image data are compressed for transmission by simple, economical circuitry. Compressor handles data in blocks of 64 samples. Mean from previous line is subtracted, and pseudo-random sequence of positive and negative 1's multiplies samples of each block. Hadamard transform applied to blocks yields 64 transform coefficients. Each coefficient is compared with approximation to corresponding coefficient of previous line, and difference is quantized. Values are transmitted or stored. Procedure is reversed to reproduce image.

Rice, R. F.↗
NASA NTRS1992

Algorithms For Compression Of Polarimetric-Radar Image Data

Two sets of algorithms provide moderate improvements in compression and decompression of scattering-matrix data from polarimetric imaging radar system. Algorithms operate on original single-look scattering matrices, do not symmetrize scattering matrices and, preserve asymmetrical data on background-noise and equipment effects.

Van Zyl, Jakob J.↗
NASA NTRS1992

Detection And Avoidance Of Obstacles By Helicopters

Report discusses problems relevant to control subsystems enabling helicopters on nap-of-the-Earth flight paths to detect and avoid obstacles automatically. Indicates similarities between this and obstacle-avoidance problem of industrial mobile robots. Two approaches extend two-dimensional obstacle-avoidance concept to three dimensions. First involves direct search of three-dimensional range-map data for indications of openings between obstacles. Second involves compression of data into two-dimensional map for path search.

Cheng, Victor H. L.↗
NASA NTRS1972

Earth resources data processor

The recent development of manned and unmanned space vehicles has brought about an almost unprecedented advance in studies concerned with remotely sensed earth observations. With this advance comes an unprecedented amount of data. The problem arises of how to efficiently analyze and compress unmanageable amounts of data into manageable amounts of useful information. A recently developed computer program is proposed as a partial solution to the above problem. The computer program is designed to determine the ground scene location and distribution of features extracted from remotely sensed earth observation data without human involvement in the data processing or a priori knowledge of ground truth. Human involvement and judgement are reserved for identification of the features presented in the compressed data.

Phillips, M. R.↗
NASA NTRS2015

A Formal Messaging Notation for Alaskan Aviation Data

Data exchange is an increasingly important aspect of the National Airspace System. While many data communication channels have become more capable of sending and receiving data at higher throughput rates, there is still a need to use communication channels efficiently with limited throughput. The limitation can be based on technological issues, financial considerations, or both. This paper provides a complete description of several important aviation weather data in Abstract Syntax Notation format. By doing so, data providers can take advantage of Abstract Syntax Notation's ability to encode data in a highly compressed format. When data such as pilot weather reports, surface weather observations, and various weather predictions are compressed in such a manner, it allows for the efficient use of throughput-limited communication channels. This paper provides details on the Abstract Syntax Notation One (ASN.1) implementation for Alaskan aviation data, and demonstrates its use on real-world aviation weather data samples as Alaska has sparse terrestrial data infrastructure and data are often sent via relatively costly satellite channels.

communications↗
DOE OSTI2021

Analyzing acoustic emission data to identify cracking modes in cement paste using an artificial neural network

This research is focused on the identification of cracking mechanisms for cement paste using acoustic emission data, recorded from compression and notched four-point bending tests. A procedure is developed for analyzing the data by employing an agglomerative hierarchical clustering method, an artificial neural network, and a ray-tracing source location algorithm. An agglomerative hierarchical clustering method is utilized to cluster the AE data from a compression test using frequency-dependent features. A neural network is trained using the compression test data and applied to the AE data emitted during the four-point bending test. The clustered data from the four-point bending test is localized using a ray-tracing algorithm. Based on the occurrence and locations of the clustered events and signal feature analyses, potential cracking mechanisms are identified and assigned.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Applying Compression-Based Metrics to Seismic Data in Support of Global Nuclear Explosion Monitoring

The analysis of seismic data for evidence of possible nuclear explosion testing is a critical global security mission that relies heavily on human expertise to identify and mark seismic signals embedded in background noise. To assist analysts in making these determinations, we adapted two compression distance metrics for use with seismic data. First, we demonstrated that the Normalized Compression Distance (NCD) metric can be adapted for use with waveform data and can identify the arrival times of seismic signals. Then we tested an approximation for the NCD called Sliding Information Distance (SLID), which can be computed much faster than NCD. We assessed the accuracy of the SLID output by comparing it to both the Akaike Information Criterion (AIC) and the judgments of expert seismic analysts. Our results indicate that SLID effectively identifies arrival times and provides analysts with useful information that can aid their analysis process.

45 MILITARY TECHNOLOGY, WEAPONRY, AND NATIONAL DEF↗
DOE OSTI2025

CODARcode/MGARD

MGARD is a software providing error-controlled lossy compression and data refactoring based on multi-grid theories. It transforms floating-point scientific data into a multilevel representation, followed by quantization and lossless encoding processes, resulting in a self-describing compressed buffer. It supports diverse data topologies, error control norms, and computing architectures.

Chen, Jieyang [University of Oregon]↗
DOE OSTI2020

Pass-efficient methods for compression of high-dimensional turbulent flow data

The future of high-performance computing, specifically on future Exascale computers, will presumably see memory capacity and bandwidth fail to keep pace with data generated, for instance, from massively parallel partial differential equation (PDE) systems. Current strategies proposed to address this bottleneck entail the omission of large fractions of data, as well as the incorporation of in situ compression algorithms to avoid overuse of memory. To ensure that post-processing operations are successful, this must be done in a way that a sufficiently accurate representation of the solution is stored. Moreover, in situations where the input/output system becomes a bottleneck in analysis, visualization, etc., or the execution of the PDE solver is expensive, the number of passes made over the data must be minimized. In the interest of addressing this problem, this work focuses on the utility of pass-efficient, parallelizable, low-rank, matrix decomposition methods in compressing high-dimensional simulation data from turbulent flows. Additionally, a particular emphasis is placed on using coarse representation of the data – compatible with the PDE discretization grid – to accelerate the construction of the low-rank factorization. This includes the presentation of a novel single-pass matrix decomposition algorithm for computing the so-called interpolative decomposition. The methods are described extensively and numerical experiments on two turbulent channel flow data are performed. In the first (unladen) channel flow case, compression factors exceeding 400 are achieved while maintaining accuracy with respect to first- and second-order flow statistics. In the particle-laden case, compression factors of 100 are achieved and the compressed data is used to recover particle velocities. These results show that these compression methods can enable efficient computation of various quantities of interest in both the carrier and disperse phases.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
NASA NTRS1989

A technique to calibrate polarimetric radar images using only image parameters and trihedral corner reflectors

The technique described uses the radar return from natural targets and at least one trihedral corner reflector to calibrate compressed polarimetric radar data for relative amplitude, relative phase, absolute amplitude, and system crosstalk. The crosstalk method is based on the theoretical result that, for natural targets with azimuthal symmetry, the copolarized and cross-polarized components of the scattering matrix are uncorrelated. This method does not require any external calibration targets to be deployed. Since compressed data are used, it is necessary to model the transmitting and receiving systems as reciprocal. The method is used to estimate the crosstalk parameters of the NASA/JPL aircraft for different types of terrain and for two frequencies. For the C-band systems the crosstalk is less than -20 dB for all ranges in the images. The crosstalk of the L-band system is a function of range, however, and may be as poor as -10 dB in the near range, leading to a noticeable distortion of the polarization signatures.

Van Zyl, Jakob J.↗
DOE OSTI2026

Maximizing efficiency of dataset compression for machine learning potentials with information theory

Machine learning interatomic potentials (MLIPs) balance high accuracy and lower costs compared to density functional theory calculations, but their performance often depends on the size and diversity of training datasets. Large datasets improve model accuracy and generalization but are computationally expensive to produce and train on, while smaller datasets risk discarding rare but important atomic environments and compromising MLIP accuracy/reliability. Here, we develop an information-theoretical framework to quantify the efficiency of dataset compression methods and propose an algorithm that maximizes this efficiency. By framing atomistic dataset compression as an instance of the minimum set cover (MSC) problem over atom-centered environments, our method identifies the smallest subset of structures that contains as much information as possible from the original dataset while pruning redundant information. The approach is extensively demonstrated on the GAP-20 and TM23 datasets and validated on 64 varied datasets from the ColabFit repository. Across all cases, MSC consistently retains outliers, preserves dataset diversity, and reproduces the long-tail distributions of forces even at high compression rates, outperforming other subsampling methods. Furthermore, MLIPs trained on MSC-compressed datasets exhibit reduced error for out-of-distribution data even in low-data regimes. We explain these results using an outlier analysis and show that such quantitative conclusions could not be achieved with conventional dimensionality reduction methods. The algorithm is implemented in the open-source QUESTS package and can be used for several tasks in atomistic modeling, from data subsampling, outlier detection, and training improved MLIPs at a lower cost.

36 MATERIALS SCIENCE↗
DOE OSTI2021

Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition

Extreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time. The challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-the-art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and ~ 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively.

Liang, Xin↗