Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data compression techniques”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8
All sourcesNASA NTRSDOE OSTI
NASA NTRS1977

Investigation of techniques for inventorying forested regions. Volume 2: Forestry information system requirements and joint use of remotely sensed and ancillary data

The author has identified the following significant results. Effects of terrain topography in mountainous forested regions on LANDSAT signals and classifier training were found to be significant. The aspect of sloping terrain relative to the sun's azimuth was the major cause of variability. A relative insolation factor could be defined which, in a single variable, represents the joint effects of slope and aspect and solar geometry on irradiance. Forest canopy reflectances were bound, both through simulation, and empirically, to have nondiffuse reflectance characteristics. Training procedures could be improved by stratifying in the space of ancillary variables and training in each stratum. Application of the Tasselled-Cap transformation for LANDSAT data acquired over forested terrain could provide a viable technique for data compression and convenient physical interpretations.

Nalepka, R. F.↗
NASA NTRS1981

Feasibility study of a microprocessor based oculometer system

The elimination of redundancy in data to maximize processing speed and minimize storage requirements were objectives in a feasibility study of a microprocessor based oculometer system that would be portable in size and flexible in use. The appropriate architectural design of the signal processor, improved optics, and the reduction of size, weight, and power to the system were investigated. A flow chart is presented showing the strategy of the design. The simulation for developing microroutines for the high speed algorithmic processor subsystem is discussed as well as the Karhunen-Loeve transform technique for data compression.

Varanasi, M. R.↗
NASA NTRS1992

Maximal codeword lengths in Huffman codes

The following question about Huffman coding, which is an important technique for compressing data from a discrete source, is considered. If p is the smallest source probability, how long, in terms of p, can the longest Huffman codeword be? It is shown that if p is in the range 0 less than p less than or equal to 1/2, and if K is the unique index such that 1/F(sub K+3) less than p less than or equal to 1/F(sub K+2), where F(sub K) denotes the Kth Fibonacci number, then the longest Huffman codeword for a source whose least probability is p is at most K, and no better bound is possible. Asymptotically, this implies the surprising fact that for small values of p, a Huffman code's longest codeword can be as much as 44 percent larger than that of the corresponding Shannon code.

Abu-Mostafa, Y. S.↗
DOE OSTI2024

Haar-Like Wavelets on Hierarchical Trees

Here, discrete wavelet methods, originally formulated in the setting of regularly sampled signals, can be adapted to data defined on a point cloud if some multiresolution structure is imposed on the cloud. A wide variety of hierarchical clustering algorithms can be used for this purpose, and the multiresolution structure obtained can be encoded by a hierarchical tree of subsets of the cloud. Prior work introduced the use of Haar-like bases defined with respect to such trees for approximation and learning tasks on unstructured data. This paper builds on that work in two directions. First, we present an algorithm for constructing Haar-like bases on general discrete hierarchical trees. Second, with an eye towards data compression, we present thresholding techniques for data defined on a point cloud with error controlled in the $L$ $\infty$ norm and in a Hölder-type norm. In a concluding trio of numerical examples, we apply our methods to compress a point cloud dataset, study the tightness of the $L$ $\infty$ error bound, and use thresholding to identify MNIST classifiers with good generalizability.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2014

Airborne Demonstration of FPGA Implementation of Fast Lossless Hyperspectral Data Compression System

Efficient on-board lossless hyperspectral data compression reduces data volume in order to meet NASA and DoD limited downlink capabilities. The technique also improves signature extraction, object recognition and feature classification capabilities by providing exact reconstructed data on constrained downlink resources. At JPL a novel, adaptive and predictive technique for lossless compression of hyperspectral data was recently developed. This technique uses an adaptive filtering method and achieves a combination of low complexity and compression effectiveness that far exceeds state-of-the-art techniques currently in use. The JPL-developed 'Fast Lossless' algorithm requires no training data or other specific information about the nature of the spectral bands for a fixed instrument dynamic range. It is of low computational complexity and thus well-suited for implementation in hardware.

data compression↗
NASA NTRS2009

Hardware Implementation of Lossless Adaptive and Scalable Hyperspectral Data Compression for Space

On-board lossless hyperspectral data compression reduces data volume in order to meet NASA and DoD limited downlink capabilities. The technique also improves signature extraction, object recognition and feature classification capabilities by providing exact reconstructed data on constrained downlink resources. At JPL a novel, adaptive and predictive technique for lossless compression of hyperspectral data was recently developed. This technique uses an adaptive filtering method and achieves a combination of low complexity and compression effectiveness that far exceeds state-of-the-art techniques currently in use. The JPL-developed 'Fast Lossless' algorithm requires no training data or other specific information about the nature of the spectral bands for a fixed instrument dynamic range. It is of low computational complexity and thus well-suited for implementation in hardware. A modified form of the algorithm that is better suited for data from pushbroom instruments is generally appropriate for flight implementation. A scalable field programmable gate array (FPGA) hardware implementation was developed. The FPGA implementation achieves a throughput performance of 58 Msamples/sec, which can be increased to over 100 Msamples/sec in a parallel implementation that uses twice the hardware resources This paper describes the hardware implementation of the 'Modified Fast Lossless' compression algorithm on an FPGA. The FPGA implementation targets the current state-of-the-art FPGAs (Xilinx Virtex IV and V families) and compresses one sample every clock cycle to provide a fast and practical real-time solution for space applications.

FPGA implementation↗
NASA NTRS2009

Fast and Adaptive Lossless On-Board Hyperspectral Data Compression System for Space Applications

Efficient on-board lossless hyperspectral data compression reduces the data volume necessary to meet NASA and DoD limited downlink capabilities. The techniques also improves signature extraction, object recognition and feature classification capabilities by providing exact reconstructed data on constrained downlink resources. At JPL a novel, adaptive and predictive technique for lossless compression of hyperspectral data was recently developed. This technique uses an adaptive filtering method and achieves a combination of low complexity and compression effectiveness that far exceeds state-of-the-art techniques currently in use. The JPL-developed 'Fast Lossless' algorithm requires no training data or other specific information about the nature of the spectral bands for a fixed instrument dynamic range. It is of low computational complexity and thus well-suited for implementation in hardware, which makes it practical for flight implementations of pushbroom instruments. A prototype of the compressor (and decompressor) of the algorithm is available in software, but this implementation may not meet speed and real-time requirements of some space applications. Hardware acceleration provides performance improvements of 10x-100x vs. the software implementation (about 1M samples/sec on a Pentium IV machine). This paper describes a hardware implementation of the JPL-developed 'Fast Lossless' compression algorithm on a Field Programmable Gate Array (FPGA). The FPGA implementation targets the current state of the art FPGAs (Xilinx Virtex IV and V families) and compresses one sample every clock cycle to provide a fast and practical real-time solution for Space applications.

image compression↗
NASA NTRS1991

Advanced techniques and technology for efficient data storage, access, and transfer

Advanced techniques for efficiently representing most forms of data are being implemented in practical hardware and software form through the joint efforts of three NASA centers. These techniques adapt to local statistical variations to continually provide near optimum code efficiency when representing data without error. Demonstrated in several earlier space applications, these techniques are the basis of initial NASA data compression standards specifications. Since the techniques clearly apply to most NASA science data, NASA invested in the development of both hardware and software implementations for general use. This investment includes high-speed single-chip very large scale integration (VLSI) coding and decoding modules as well as machine-transferrable software routines. The hardware chips were tested in the laboratory at data rates as high as 700 Mbits/s. A coding module's definition includes a predictive preprocessing stage and a powerful adaptive coding stage. The function of the preprocessor is to optimally process incoming data into a standard form data source that the second stage can handle.The built-in preprocessor of the VLSI coder chips is ideal for high-speed sampled data applications such as imaging and high-quality audio, but additionally, the second stage adaptive coder can be used separately with any source that can be externally preprocessed into the 'standard form'. This generic functionality assures that the applicability of these techniques and their recent high-speed implementations should be equally broad outside of NASA.

Rice, Robert F.↗
NASA NTRS2005

Low-Complexity Lossless Compression of Hyperspectral Imagery via Adaptive Filtering

A low-complexity, adaptive predictive technique for lossless compression of hyperspectral data is presented. The technique relies on the sign algorithm from the repertoire of adaptive filtering. The compression effectiveness obtained with the technique is competitive with that of the best of previously described techniques with similar complexity.

Klimesh, M.↗
NASA NTRSDate not supplied

Compressive Sensing Based Data Acquisition Architecture for Transient Stellar Events in Crowded Star Fields

Compressive sensing is a mathematical technique for simultaneous data acquisition and compression. In this work, we show a CS based architecture for acquiring and reconstructing transient stellar events. This architecture reconstructs a differenced image itself, eliminating the need for any sparse domain transforms, otherwise required for traditional CS reconstruction. The resulting reconstructed differenced image is of critical importance as the information required for generating a time-series photometric light curve is obtained only from a differenced image. Hence, reconstructing a crowded star spatial image, followed by differencing is wasteful. This architecture eliminates the need to 1.) transform an image to a sparse domain, 2.) Reconstruct a dense field, and then apply differencing on the image to obtain the image of critical value. We study the case of microlensing to depict a star source experiencing magnification in time. Our results show that this architecture is able to reconstruct star source magnitudes with magnification factors greater than 1 for clean images with error less than 2% using only 10% of the Nyquist rate samples.

Asmita Korde-Patel↗
NASA NTRSDate not supplied

Compressive Sensing Based Data Acquisition Architecture for Transient Stellar Events in Crowded Star Fields

Compressive sensing is a mathematical technique for simultaneous data acquisition and compression. In this work, we show a CS based architecture for acquiring and reconstructing transient stellar events. This architecture recon-structs a differenced image itself, eliminating the need for any sparse domain transforms, otherwise required for traditional CS reconstruction. The resulting reconstructed differenced image is of critical importance as the information required for generating a time-series photometric light curve is obtained only from a differenced image. Hence, reconstructing a crowded star spatial image, followed by differencing is wasteful. This architecture eliminates the need to 1.) transform an image to a sparse domain, 2.) Reconstruct a dense field, and then apply differencing on the image to obtain the image of critical value. We study the case of microlensing to depict a star source experiencing magnification in time. Our results show that this architecture is able to reconstruct star source magnitudes with magnification factors greater than 1 for clean images with error less than 2% using only 10% of the Nyquist rate samples.

Compressive Sensing, Data acquisition, Image diffe↗
NASA NTRSDate not supplied

Compressive Sensing Based Data Acquisition Architecture for Transient Stellar Events in Crowded Star Fields

Compressive sensing is a mathematical technique for simultaneous data acquisition and compression. In this work, we show a CS based architecture for acquiring and reconstructing transient stellar events. This architecture recon-structs a differenced image itself, eliminating the need for any sparse domain transforms, otherwise required for traditional CS reconstruction. The resulting reconstructed differenced image is of critical importance as the information required for generating a time-series photometric light curve is obtained only from a differenced image. Hence, reconstructing a crowded star spatial image, followed by differencing is wasteful. This architecture eliminates the need to 1.) transform an image to a sparse domain, 2.) Reconstruct a dense field, and then apply differencing on the image to obtain the image of critical value. We study the case of microlensing to depict a star source experiencing magnification in time. Our results show that this architecture is able to reconstruct star source magnitudes with magnification factors greater than 1 for clean images with error less than 2% using only 10% of the Nyquist rate samples.

Asmita Korde-patel↗
NASA NTRS2022

Compressive Sensing Based Space Flight Instrument Constellation for Measuring Gravitational Microlensing Parallax

In this work, we provide a compressive sensing architecture for implementing on a space based observatory for detecting transient photometric parallax caused by gravitational microlensing events. Compressive sensing (CS) is a simultaneous data acquisition and compression technique, which can greatly reduce on-board resources required for space flight data storage and ground transmission. We simulate microlensing parallax observations using a space observatory constellation, based on CS detectors. Our results show that average CS error is less than 0.5% using 25% Nyquist rate samples. The error at peak magnification time is significantly lower than the error for distinguishing any two microlensing parallax curves at their peak magnification. Thus, CS is an enabling technology for detecting microlensing parallax, without causing any loss in detection accuracy.

compressive sensing↗
DOE OSTI2023

A General Framework for Progressive Data Compression and Retrieval

In scientific simulations, observations, and experiments, the transfer of data to and from disk and across networks has become a major bottleneck for data analysis and visualization. Compression techniques have been employed to tackle this challenge, but traditional lossy methods often demand conservative error tolerances to meet the numerical accuracy requirements of both anticipated and unknown data analysis tasks. Progressive data compression and retrieval has emerged as a promising solution, where each analysis task dictates its own accuracy needs. However, few analysis algorithms inherently support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. Here, this paper presents a framework that enables progressive-precision data queries for any data compressor or numerical representation. Our strategy hinges on a multi-component representation that successively reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented this approach with four established scientific data compressors and assessed its effectiveness using real-world data sets from the SDRBench collection. The results show that our framework competes in accuracy with the standalone compressors it is based upon. Additionally, (de)compression time is proportional to the number of components requested by the user. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1989

Data compression experiments with LANDSAT thematic mapper and Nimbus-7 coastal zone color scanner data

A case study is presented where an image segmentation based compression technique is applied to LANDSAT Thematic Mapper (TM) and Nimbus-7 Coastal Zone Color Scanner (CZCS) data. The compression technique, called Spatially Constrained Clustering (SCC), can be regarded as an adaptive vector quantization approach. The SCC can be applied to either single or multiple spectral bands of image data. The segmented image resulting from SCC is encoded in small rectangular blocks, with the codebook varying from block to block. Lossless compression potential (LDP) of sample TM and CZCS images are evaluated. For the TM test image, the LCP is 2.79. For the CZCS test image the LCP is 1.89, even though when only a cloud-free section of the image is considered the LCP increases to 3.48. Examples of compressed images are shown at several compression ratios ranging from 4 to 15. In the case of TM data, the compressed data are classified using the Bayes' classifier. The results show an improvement in the similarity between the classification results and ground truth when compressed data are used, thus showing that compression is, in fact, a useful first step in the analysis.

Tilton, James C.↗
NASA NTRS1990

Remote sensing science for the Nineties; Proceedings of IGARSS '90 - 10th Annual International Geoscience and Remote Sensing Symposium, University of Maryland, College Park, May 20-24, 1990. Vols. 1, 2, & 3

Various papers on remote sensing (RS) for the nineties are presented. The general topics addressed include: subsurface methods, radar scattering, oceanography, microwave models, atmospheric correction, passive microwave systems, RS in tropical forests, moderate resolution land analysis, SAR geometry and SNR improvement, image analysis, inversion and signal processing for geoscience, surface scattering, rain measurements, sensor calibration, wind measurements, terrestrial ecology, agriculture, geometric registration, subsurface sediment geology, radar modulation mechanisms, radar ocean scattering, SAR calibration, airborne radar systems, water vapor retrieval, forest ecosystem dynamics, land analysis, multisensor data fusion. Also considered are: geologic RS, RS sensor optical measurements, RS of snow, temperature retrieval, vegetation structure, global change, artificial intelligence, SAR processing techniques, geologic RS field experiment, stochastic modeling, topography and Digital Elevation model, SAR ocean waves, spaceborne lidar and optical, sea ice field measurements, millimeter waves, advanced spectroscopy, spatial analysis and data compression, SAR polarimetry techniques. Also discussed are: plant canopy modeling, optical RS techniques, optical and IR oceanography, soil moisture, sea ice back scattering, lightning cloud measurements, spatial textural analysis, SAR systems and techniques, active microwave sensing, lidar and optical, radar scatterometry, RS of estuaries, vegetation modeling, RS systems, EOS/SAR Alaska, applications for developing countries, SAR speckle and texture.

Source record↗
DOE OSTI2025

Hybrid learning techniques for scientific data reduction with performance guarantees

The research initiatives supported by the U.S. Department of Energy (DOE) Grant DE-SC0022265 are fundamentally aimed at pioneering advanced machine learning (ML) techniques for scientific data compression within high-performance computing (HPC) environments. This comprehensive body of work addresses the critical challenge posed by the exponential growth of data generated by scientific simulations in domains such as fusion energy, climate modeling, and computational fluid dynamics (CFD). A core objective is to develop compression algorithms that achieve substantial data reduction—often by orders of magnitude—while rigorously ensuring the fidelity of both the primary data (PD) and scientifically crucial derived quantities of interest (QoI). The methodologies deployed under this grant integrate sophisticated deep learning architectures, prominently featuring autoencoders, advanced generative models like conditional diffusion, and hybrid learning techniques. Key innovations include the development of Guaranteed Autoencoders (GAE) and the Guaranteed Conditional Diffusion with Tensor Correction (GCDTC) framework, which provide explicit, instance-level error bounds on reconstructed data. Furthermore, specialized strategies such as nonlinear constraint satisfaction are employed to preserve the integrity of QoI, a vital requirement for the trustworthiness of downstream scientific analyses. This research also focuses on the design and implementation of scalable, GPU-accelerated software pipelines that seamlessly integrate into existing HPC workflows, ensuring both computational efficiency and practical applicability. The CAESAR framework, for example, unifies foundation and generative models to create an adaptive and efficient compression solution for spatio-temporal scientific data. Collectively, these efforts represent a significant advancement in mitigating the scientific data deluge, enabling more effective data management, accelerated scientific discovery, and optimized utilization of HPC resources.

97 MATHEMATICS AND COMPUTING↗