SEARCH · Engineering Papers

Results for “Lossy compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

SourceResearch connectionFrom yearThrough year

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 145 records · Page 8

All sources NASA NTRS DOE OSTI

DOE OSTI2024

Haar-Like Wavelets on Hierarchical Trees

Here, discrete wavelet methods, originally formulated in the setting of regularly sampled signals, can be adapted to data defined on a point cloud if some multiresolution structure is imposed on the cloud. A wide variety of hierarchical clustering algorithms can be used for this purpose, and the multiresolution structure obtained can be encoded by a hierarchical tree of subsets of the cloud. Prior work introduced the use of Haar-like bases defined with respect to such trees for approximation and learning tasks on unstructured data. This paper builds on that work in two directions. First, we present an algorithm for constructing Haar-like bases on general discrete hierarchical trees. Second, with an eye towards data compression, we present thresholding techniques for data defined on a point cloud with error controlled in the $L$ $\infty$ norm and in a Hölder-type norm. In a concluding trio of numerical examples, we apply our methods to compress a point cloud dataset, study the tightness of the $L$ $\infty$ error bound, and use thresholding to identify MNIST classifiers with good generalizability.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2022

Data Compression in the NEXT-100 Data Acquisition System

NEXT collaboration detectors are based on energy measured by an array of photomultipliers (PMT) and topological event filtering based on an array of silicon photomultipliers (SiPMs). The readout of the PMT sensors for low-frequency noise effects and detector safety issues requires a grounded cathode connection that makes the readout AC-couple with variations in the signal baseline. Strict detector requirements of energy resolution better than 1% FWHM require a precise baseline reconstruction that is performed offline for data analysis and detector performance characterization. Baseline variations make it inefficient to apply traditional lossy data compression techniques, such as zero-suppression, that help to minimize data throughput and, therefore, the dead time of the system. However, for the readout of the SiPM sensors with less demanding requirements in terms of accuracy, a traditional zero-suppression is currently applied with a configuration that allows for a compression ratio of around 71%. The third stage in the NEXT detectors program, the NEXT-100 detector, is a 100 kg detector that instruments approximately five times more PMT sensors and twice the number of SiPM sensors than its predecessor, the NEXT-White detector, putting more pressure in the DAQ throughput, expected to be over 900 MB/s with the current configuration, which will worsen the dead time of the acquisition data system. This paper describes the data compression techniques applied to the sensor data in the NEXT-100 detector, which reduces data throughput and minimizes dead time while maintaining the event rate to the level of its predecessor, around 50 Hz.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

DOE OSTI2020

Multilevel Techniques for Compression and Reduction of Scientific Data---The Unstructured Case

Previous work on multilevel techniques for compression and reduction of scientific data is extended to the case of data given on unstructured meshes in two and three dimensions. The centerpiece of the work is a decomposition algorithm which is shown to be optimal, in terms of both storage and operational complexity, applicable to unstructured grids in both two and three dimensions, and which implicitly gives a Riesz basis that can be exploited to reduce the data while maintaining rigorous bounds on the loss incurred. The flexibility of the approach is illustrated by applications to potential flow around an airfoil and the effect of compression on quantities of interest relevant to airfoil design; compression of computational simulation of a nonlinear reaction-diffusion system with special attention given to the problem of time series reduction; and, data from a simulation of magnetically confined plasma in a fusion reactor reduced so as to preserve the electric field computed from the data.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1998

Pre-coding method and apparatus for multiple source or time-shifted single source data and corresponding inverse post-decoding method and apparatus

A pre-coding method and device for improving data compression performance by removing correlation between a first original data set and a second original data set, each having M members, respectively. The pre-coding method produces a compression-efficiency-enhancing double-difference data set. The method and device produce a double-difference data set, i.e., an adjacent-delta calculation performed on a cross-delta data set or a cross-delta calculation performed on two adjacent-delta data sets, from either one of (1) two adjacent spectral bands coming from two discrete sources, respectively, or (2) two time-shifted data sets coming from a single source. The resulting double-difference data set is then coded using either a distortionless data encoding scheme (entropy encoding) or a lossy data compression scheme. Also, a post-decoding method and device for recovering a second original data set having been represented by such a double-difference data set.

Yeh, Pen-Shu↗

NASA NTRS1997

Pre-coding method and apparatus for multiple source or time-shifted single source data and corresponding inverse post-decoding method and apparatus

Yeh, Pen-Shu↗

DOE OSTI2024

Feasibility of Formulating Ecosystem Biogeochemical Models From Established Physical Rules

Abstract To improve the predictive capability of ecosystem biogeochemical models (EBMs), we discuss the feasibility of formulating biogeochemical processes using physical rules that have underpinned the many successes in computational physics and chemistry. We argue that the currently popular empirically based approaches, such as multiplicative empirical response functions and the law of the minimum, will not lead to EBM formulations that can be continuously refined to incorporate improved mechanistic understanding and empirical observations of biogeochemical processes. Instead, we propose that EBM parameterizations, as a lossy data compression problem, can be better formulated using established physical rules widely used in computational physics and chemistry, and different biogeochemical processes can be more robustly integrated within a reactive‐transport framework. Through several examples, we demonstrate how mathematical representations derived from physical rules can improve understanding of relevant biogeochemical processes and enable more effective communication between modelers, observationalists, and experimentalists regarding essential questions, such as what measurements are needed to meaningfully inform models and how can models generate new process‐level hypotheses to test in empirical studies. Finally, while empirical models with more parameters are often less robust, physical rules‐based models can be more robust and show lower predictive equifinality, stemming from their enhanced consistency in representations of processes, interactions and spatial scaling.

54 ENVIRONMENTAL SCIENCES↗

DOE OSTI2025

Variational autoencoders for at-source data reduction and anomaly detection in high energy particle detectors

Detectors in next-generation high-energy physics experiments face several daunting requirements, such as high data rates, damaging radiation exposure, and stringent constraints on power, space, and latency. To address these challenges, machine learning in readout electronics can be leveraged for smart detector designs, enabling intelligent inference and data reduction at-source. Variational autoencoders (VAEs) offer a variety of benefits for front-end readout; an on-sensor encoder can perform efficient lossy data compression while simultaneously providing a latent space representation that can be used for anomaly detection. Results are presented from low-latency and resource-efficient VAEs for front-end data processing in a futuristic silicon pixel detector. Encoder-based data compression is found to preserve good performance of off-detector analysis while significantly reducing the off-detector data rate as compared to a similarly sized data filtering approach. Furthermore, the latent space information is found to be a useful discriminator in the context of real-time sensor defect monitoring. Together, these results highlight the multifaceted utility of autoencoder-based front-end readout schemes and motivate their consideration in future detector designs.

47 OTHER INSTRUMENTATION↗

NASA NTRS1968

Characteristic waves on an infinite cylindrical antenna in a plasma medium.

Current nature on infinite cylindrical antenna in compressible anisotropic lossy plasma medium, noting characteristic waves on infinite cylinder

Miller, E. K.↗

NASA NTRS1989

Imaging spectrometry - Technology and applications

The development history and current status of NASA imaging-spectrometer (IS) technology are discussed in a review covering the period 1982-1988. Consideration is given to the Airborne IS first flown in 1982, the second-generation Airborne Visible and IR IS (AVIRIS), the High-Resolution IS being developed for the EOS polar platform, improved two-dimensional focal-plane arrays for the short-wave IR spectral region, and noncollinear acoustooptic tunable filters for use as spectral dispersing elements. Also examined are approaches to solving the data-processing problems posed by the high data volumes of state-of-the-art ISs (e.g., 160 MB per 600 x 600-pixel AVIRIS scene), including intelligent data editing, lossless and lossy data compression techniques, and direct extraction of scientifically meaningful geophysical and biophysical parameters.

Solomon, Jerry E.↗

NASA NTRS1989

Hybrid LZW compression

The Science Data Management and Science Payload Operations subpanel reports from the NASA Conference on Scientific Data Compression (Snowbird, Utah in 1988) indicate the need for both lossless and lossy image data compression systems. The ranges developed by the subpanel suggest ratios of 2:1 to 4:1 for lossless coding and 2:1 to 6:1 for lossy predictive coding. For the NASA Freedom Science Video Processing Facility it would be highly desirable to implement one baseline compression system which would meet both of these criteria. Presented here is such a system, utilizing an LZW hybrid coding scheme which is adaptable to either type of compression. Simulation results are presented with the hybrid LZW algorithm operating in each of its modes.

Lewis, H. Garton, Jr.↗

NASA NTRS1992

Efficient transfer of images over networks

Effective remote observing requires sending large images over long distances. The usual approach to the transfer problem is to require high bandwidth transmission links, which are expensive to install and operate. An alternative approach is to use existing low-bandwidth connections, such as phone lines or the Internet, in a highly efficient manner by compressing the images. The combined use of existing low-cost infrastructure and standard networking software means that remote observing can be made practical even for small observatories with limited network resources. The authors have implemented such a scheme based on the H-transform compression method developed for astronomical images, which are often resistant to compression because they are noisy. The H-transform can be used for either lossy or lossless compression, and compression factors of at least 10 can be achieved with no noticeable losses in the astrometric or photometric properties of the compressed images. The H-transform allows us to organize the information in an image so that the 'useful' information can be sent first, followed by the noise, which makes up the bulk of the transmission. The receiver can invert a partially received set of H-coefficients, creating an image that improves with time. The H-transform is particularly well-suited to this style of incremental reconstruction, because the spatially localized nature of the basis functions of the H-transorm prevents the appearance of artifacts such as ringing around point sources and edges. The authors' implementation uses the WIYN Telescope Control System's TCP-based communications protocol. An 800x800 16-bit astronomical image was sent over a 2400 baud connection, which would normally take about 71 minutes; after only 60 seconds, the partially received H-transform produced an image that did not differ appreciably from the original. This poster presents a quantification of the efficiencies, as well as examples of images reconstructed from partial data.

Percival, J. W.↗

NASA NTRS2011

Analysis of Compression Algorithm in Ground Collision Avoidance Systems (Auto-GCAS)

Automatic Ground Collision Avoidance Systems (Auto-GCAS) utilizes Digital Terrain Elevation Data (DTED) stored onboard a plane to determine potential recovery maneuvers. Because of the current limitations of computer hardware on military airplanes such as the F-22 and F-35, the DTED must be compressed through a lossy technique called binary-tree tip-tilt. The purpose of this study is to determine the accuracy of the compressed data with respect to the original DTED. This study is mainly interested in the magnitude of the error between the two as well as the overall distribution of the errors throughout the DTED. By understanding how the errors of the compression technique are affected by various factors (topography, density of sampling points, sub-sampling techniques, etc.), modifications can be made to the compression technique resulting in better accuracy. This, in turn, would minimize unnecessary activation of A-GCAS during flight as well as maximizing its contribution to fighter safety.

Schmalz, Tyler↗

NASA NTRS2001

Ocean-Science Mission Needs: Real-Time AUV Data for Command, Control, and Model Inputs

Predictive models for tides, hydrodynamics, and bio-optical properties affecting the visibility and buoyancy of coastal waters are needed to evaluate the safety of personnel and equipment engaged in maritime operations under potentially hazardous conditions. Predicted currents can be markedly different for two-layer systems affected by terrestrial runoff than for well-mixed conditions because the layering decouples the surface and bottom Ekman layers and rectifies the current response to oscillatory upwelling-and downwelling-favorable winds. Standard ocean models (e.g. Princeton Ocean Model) require initial-and boundary data on the physical and optical properties of the multilayered water column to provide accurate simulations of heat budgets and circulation. Two observational systems are designed to measure vertically structured conditions on the West Florida Shelf (WFS): a tethered buoy network and an autonomous underwater vehicle (AUV) observational system. The AUV system is described with a focus on the observational systems that challenge or limit the communications command and control network for various types of measurement programs. These include vertical oscillatory missions on shelf transects to observe the optical and hydrographic properties of the water column, and bottom-following missions for measuring the bottom albedo. Models of light propagation, absorption, and conversion to heat as well as determination of the buoyancy terms for physical models require these measurements. High data rates associated with video bottom imagery are the most challenging for the real-time, command and control communications system, but they are met through a combination of loss-less and lossy data-compression methods, depending upon the data-rate of the radio links.

Carder, Kendall L.↗

NASA NTRSDate not supplied

Assimilation of Reconstructed Radiances from IASI Principal Component Scores into the GEOS-ADAS

Hyperspectral Infrared sounders such as IASI, AIRS, and CrIS have long been an integral part of radiance assimilation in numerical weather prediction (NWP), providing vertical profiles of water vapor and temperature information. Principal Component Scores (PCS) are a lossy form of compression that retains most information, such as temperature and moisture, by using a large training set of atmospheric profiles. However, PCS may not well represent profiles which are rare events, such as volcanic eruptions, and drops some sources of random noise. There has been an increased interest in the use of PCS as EUMETSAT plans to distribute future geostationary sounder radiances from MTG-IRS via PCS only. NWP centers use two approaches to deal with PCS: direct assimilation of the PCS by modifying the radiative transfer model to produce PCS and the associated Jacobians, or a simpler approach of decompressing the PCS and reconstructing the radiances back into channel space to allow assimilating radiances without modifications to the data assimilation system. EUMETSAT has developed a PCS product for IASI that has been operational since 2011. We utilize this product opting for the simpler approach, decompressing IASI PCS into channel space, and assimilating those radiances using the GEOS-ADAS. We then compare this with a control using the standard IASI radiance product. Resulting differences in global forecast statistics, differences in Forecast Sensitivity to Observation Impact, along with implications for implementation and quality control are discussed.

Bryan M. Karpowicz↗

DOE OSTI2023

A General Framework for Progressive Data Compression and Retrieval

In scientific simulations, observations, and experiments, the transfer of data to and from disk and across networks has become a major bottleneck for data analysis and visualization. Compression techniques have been employed to tackle this challenge, but traditional lossy methods often demand conservative error tolerances to meet the numerical accuracy requirements of both anticipated and unknown data analysis tasks. Progressive data compression and retrieval has emerged as a promising solution, where each analysis task dictates its own accuracy needs. However, few analysis algorithms inherently support progressive data processing, and adapting compression techniques, file formats, client/server frameworks, and APIs to support progressivity can be challenging. Here, this paper presents a framework that enables progressive-precision data queries for any data compressor or numerical representation. Our strategy hinges on a multi-component representation that successively reduces the error between the original and compressed field, allowing each field in the progressive sequence to be expressed as a partial sum of components. We have implemented this approach with four established scientific data compressors and assessed its effectiveness using real-world data sets from the SDRBench collection. The results show that our framework competes in accuracy with the standalone compressors it is based upon. Additionally, (de)compression time is proportional to the number of components requested by the user. Finally, our framework allows for fully lossless compression using lossy compressors when a sufficient number of components are employed.

97 MATHEMATICS AND COMPUTING↗

NASA NTRS1994

Performance considerations for the application of the lossless browse and residual model

A hybrid lossless compression model employing both the (lossy) JPEG DCT algorithm and one of a selection of lossless image compression methods has been tested. The hybrid model decomposes the original image into a low-loss quick-look browse and a residual image. The lossless compression methods tested in the model are Huffman, arithmetic, LZW, lossless JPEG, and diagonal coding. For both the direct and the hybrid application of these lossless methods, the compression ratios (CR's) are calculated and compared on three test images. For each lossless method tested, the hybrid model had no more than a nominal loss in compression efficiency relative to the direct approach. In many cases, the hybrid model provided a significant compression gain. When used in the hybrid model, lossless JPEG outperformed the other lossless methods over a broad range of browse image qualities.

Abbott, Walter D., III↗

NASA NTRS2010

Context Modeler for Wavelet Compression of Spectral Hyperspectral Images

A context-modeling sub-algorithm has been developed as part of an algorithm that effects three-dimensional (3D) wavelet-based compression of hyperspectral image data. The context-modeling subalgorithm, hereafter denoted the context modeler, provides estimates of probability distributions of wavelet-transformed data being encoded. These estimates are utilized by an entropy coding subalgorithm that is another major component of the compression algorithm. The estimates make it possible to compress the image data more effectively than would otherwise be possible. The following background discussion is prerequisite to a meaningful summary of the context modeler. This discussion is presented relative to ICER-3D, which is the name attached to a particular compression algorithm and the software that implements it. The ICER-3D software is summarized briefly in the preceding article, ICER-3D Hyperspectral Image Compression Software (NPO-43238). Some aspects of this algorithm were previously described, in a slightly more general context than the ICER-3D software, in "Improving 3D Wavelet-Based Compression of Hyperspectral Images" (NPO-41381), NASA Tech Briefs, Vol. 33, No. 3 (March 2009), page 7a. In turn, ICER-3D is a product of generalization of ICER, another previously reported algorithm and computer program that can perform both lossless and lossy wavelet-based compression and decompression of gray-scale-image data. In ICER-3D, hyperspectral image data are decomposed using a 3D discrete wavelet transform (DWT). Following wavelet decomposition, mean values are subtracted from spatial planes of spatially low-pass subbands prior to encoding. The resulting data are converted to sign-magnitude form and compressed. In ICER-3D, compression is progressive, in that compressed information is ordered so that as more of the compressed data stream is received, successive reconstructions of the hyperspectral image data are of successively higher overall fidelity.

Kiely, Aaron↗

NASA NTRS1994

Automatic commanding of the Mars Observer Camera

Mars Observer, launched in September 1992, was intended to be a 'survey-type' mission that acquired global coverage of Mars from a low, circular, near-polar orbit during an entire Martian year. As such, most of its instruments had fixed data rates, wide fields of view, and relatively low resolution, with fairly limited requirements for commanding. An exception is the Mars Observer Camera, or MOC. The MOC consists of a two-color Wide Angle (WA) system that can acquire both global images at low resolution (7.5 km/pixel) and regional images at commandable resolutions up to 250 m/pixel. Complementing the WA is the Narrow Angle (NA) system, that can acquire images at 8 resolutions from 12 m/pixel to 1.5 m/pixel, with a maximum crosstrack dimension of 3 km. The MOC also provides various forms of data compression (both lossless and lossy), and is designed to work at data rates from 700 bits per second (bps) to over 80k bps. Because of this flexibility, developing MOC command sequences is much more difficult than the routine mode-changing that characterizes other instrument operations. Although the MOC cannot be pointed (the spacecraft is fixed nadir-pointing and has no scan platform), the timing, downlink stream allocation, compression type and parameters, and image dimensions of each image must be commanded from the ground, subject to the constraints inherent in the MOC and the spacecraft. To minimize the need for a large operations staff, the entire command generation process has been automated within the MOC Ground Data System. Following the loss of the Mars Observer spacecraft in August 1993, NASA intends to launch a new spacecraft, Mars Global Surveyor (MGS), in late 1996. This spacecraft will carry the MOC flight spare (MOC 2). The MOC 2 operations plan will be largely identical to that developed for MOC, and all of the algorithms described here are applicable to it.

Caplinger, Michael↗