Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 595 records · Page 33

Multi-layer holographic bifurcative neural network system for real-time adaptive EOS data analysis

Optical data processing techniques have the inherent advantage of high data throughout, low weight and low power requirements. These features are particularly desirable for onboard spacecraft in-situ real-time data analysis and data compression applications. the proposed multi-layer optical holographic neural net pattern recognition technique will utilize the nonlinear photorefractive devices for real-time adaptive learning to classify input data content and recognize unexpected features. Information can be stored either in analog or digital form in a nonlinear photofractive device. The recording can be accomplished in time scales ranging from milliseconds to microseconds. When a system consisting of these devices is organized in a multi-layer structure, a feedforward neural net with bifurcating data classification capability is formed. The interdisciplinary research will involve the collaboration with top digital computer architecture experts at the University of Southern California.

Liu, Hua-Kuang↗

Digital Motion Imagery, Interoperability Challenges for Space Operations

With advances in available bandwidth from spacecraft and between terrestrial control centers, digital motion imagery and video is becoming more practical as a data gathering tool for science and engineering, as well as for sharing missions with the public. The digital motion imagery and video industry has done a good job of creating standards for compression, distribution, and physical interfaces. Compressed data streams can easily be transmitted or distributed over radio frequency, internet protocol, and other data networks. All of these standards, however, can make sharing video between spacecraft and terrestrial control centers a frustrating and complicated task when different standards and protocols are used by different agencies. This paper will explore the challenges presented by the abundance of motion imagery and video standards, interfaces and protocols with suggestions for common formats that could simplify interoperability between spacecraft and ground support systems. Real-world examples from the International Space Station will be examined. The paper will also discuss recent trends in the development of new video compression algorithms, as well likely expanded use of Delay (or Disruption) Tolerant Networking nodes.

Grubbs, Rodney↗

Pioneer Venus Sounder Probe Neutral Gas Mass Spectrometer

A neutral gas mass spectrometer was flown to Venus as part of the Pioneer Venus Multiprobe to measure the composition of its lower atmosphere. The instrument, mounted in the Sounder Probe, was activated after the probe entered the top of the atmosphere, and it obtained data during the descent from 62 km to the surface. Atmospheric gases were sampled through a pair of microleaks, the effluent from which was pumped by a combination of ion and getter pumping. A pneumatically operated valve, controlled by the ambient atmospheric pressure, maintained the ion source pressure at a nearly constant value during descent while the atmospheric pressure varied by three orders of magnitude. A single focusing magnetic sector field mass spectrometer with mass resolution sufficient to reasonably separate argon from C3H4 at 40 amu provided the mass analysis and relative abundance measurements. A microprocessor controlled the operation of the mass spectrometer through a highly efficient peak-tip stepping routine and data compression algorithm that effected a scan of the mass spectrum from 1 to 208 amu in 64 sec while requiring an information rate of only 40 bits/sec to return the data to earth. A subscale height altitude resolution was thus obtained. Weight, size, and power requirements were minimized to be consistent with interplanetary flight contraints.

Hoffman, J. H.↗

A Tailored Convolutional Neural Network for Nonlinear Manifold Learning of Computational Physics Data Using Unstructured Spatial Discretizations

In this work, we propose a nonlinear manifold learning technique based on deep convolutional autoencoders that is appropriate for model order reduction of physical systems in complex geometries. Convolutional neural networks have proven to be highly advantageous for compressing data arising from systems demonstrating a slow-decaying Kolmogorov n-width. However, these networks are restricted to data on structured meshes. Unstructured meshes are often required for performing analyses of real systems with complex geometry. Our custom graph convolution operators based on the available differential operators for a given spatial discretization effectively extend the application space of deep convolutional autoencoders to systems with arbitrarily complex geometry that are typically discretized using unstructured meshes. We propose sets of convolution operators based on the spatial derivative operators for the underlying spatial discretization, making the method particularly well suited to data arising from the solution of partial differential equations. We demonstrate the method using examples from heat transfer and fluid mechanics and show better than an order of magnitude improvement in accuracy over linear methods.

97 MATHEMATICS AND COMPUTING↗

Information theoretical assessment of digital imaging systems

The end-to-end performance of image gathering, coding, and restoration as a whole is considered. This approach is based on the pivotal relationship that exists between the spectral information density of the transmitted signal and the restorability of images from this signal. The information-theoretical assessment accounts for (1) the information density and efficiency of the acquired signal as a function of the image-gathering system design and the radiance-field statistics, and (2) the improvement in information efficiency and data compression that can be gained by combining image gathering with coding to reduce the signal redundancy and irrelevancy. It is concluded that images can be restored with better quality and from fewer data as the information efficiency of the data is increased. The restoration correctly explains the image gathering and coding processes and effectively suppresses the image-display degradations.

John, Sarah↗

LUNA Condition-Based Monitoring Update: Auto-encoders on Actuator Data [Slides]

An auto-encoder is a neural network trained to output a reconstruction of its input. There is a ‘bottleneck’ layer whose dimension is typically lower than that of the input. Because the input passes through the bottleneck, the auto-encoder must learn to compress (encode) the input into this low-dimensional form, and to decompress (decode) the compressed data into a close approximation of its original form.

97 MATHEMATICS AND COMPUTING↗

QProR: An Efficient Framework for Quantity-of-Interest Based Progressive Retrieval with Guaranteed Error Control

Scientific applications generate an unprecedented volume of data, overwhelming the network and file systems’ bandwidth and posing challenges for efficient and scalable data retrieval and analysis. Progressive data compression offers a promising solution by enabling on-demand retrieval at reduced size. However, existing progressive methods either fail to bound the errors in essential quantities of interest (QoIs) derived from raw data or suffer from suboptimal retrieval efficiency. In this work, we propose QProR, an efficient QoI-based progressive framework that optimizes progressive retrieval for target QoIs. Our key contributions include: (1) a systematic framework that integrates error-controlled lossy compressors with bitplane encoding while decoupling the two processes for high flexibility and adaptability; (2) a novel weighted bitplane encoding method which incorperates QoI knowledge into data refactoring to enhance retrieval efficiency; (3) an optimized retrieval strategy that accounts for the varying impacts of different variables on multivariate QoIs; (4) comprehensive evaluations using six real-world datasets from multiple scientific applications and thorough comparisons against state of the arts. Experimental results demonstrate that QProR achieves up to 80.38% reduction in the retrieval size under the same requested QoI error tolerance, when compared with the best-performing existing methods. When transferring 384 GB of scientific data to remote sites, QProR delivers up to 1.68 × speedup in the end-to-end data transfer performance.

Li, Wenbo [University of Kentucky]↗

Information preserving coding for multispectral data

A general formulation of the data compression system is presented. A method of instantaneous expansion of quantization levels by reserving two codewords in the codebook to perform a folding over in quantization is implemented for error free coding of data with incomplete knowledge of the probability density function. Results for simple DPCM with folding and an adaptive transform coding technique followed by a DPCM technique are compared using ERTS-1 data.

Duan, J. R.↗

Source encoding for orbiter communications links

The feasibility of using data compression to improve link efficiency as an alternative to increased transmitter power, reducing receiver noise figures, increasing antenna gain through more stringent orbiter attitude constraints, etc. is studied. A method of encoding digital data is developed which permits low band-width encoding as well as a unique system of adaptive run length encoding. The effectiveness of these techniques for the air-to-ground link and for the bandwidth-limited ground-to-ground data link used for the orbiter downlink data is evaluated. Results are presented.

Source record↗

Image recovery from edge primitives

A method for extracting edge primitives from Mach-band patterns is presented together with a method for recovering image representations of features outlined by the edge boundaries. The accuracy, stability, and resolution of these representations are assessed. Since these representations are most commonly used in characterizing targets, this method of low-level processing offers new opportunities for computer vision and high data-compressing coding. Two bandpass filters are considered, the spatially invariant Laplacian of Gaussian filter and spatially variant intensity-dependent spatial (IDS) summation. It is shown that the recovery from the IDS bandpass data is particularly advantageous in applications for which robustness to local and temporal variations in illumination is important. It is concluded that the edge primitives extracted from bandpassed images can be an efficient way to store, transmit, and represent images.

Alter-Gartenberg, Rachel↗

SAMPA Based Streaming Readout Data Acquisition Prototype

We have assembled a small-scale streaming data acquisition system based on the SAMPA front-end ASIC. The 32-channel SAMPA chip was designed for the high-luminosity upgrade of the ALICE Time Projection Chamber (TPC) detector at the CERN Large Hadron Collider. The goals of the prototype system are to determine if the SAMPA chip is appropriate for use in detector systems at Jefferson Lab, and to gain experience with the hardware and software required to deploy streaming data acquisition systems in nuclear physics experiments. The 800 channel system is composed of components used in the ALICE TPC data acquisition upgrade. Five front-end cards (FEC) support five SAMPA chips each. SAMPA data streams on an FEC are concentrated into two high-speed (4.48 Gb/s) serial data streams by a pair Gigabit Transceiver ASICs (GBTx). These ten streams (44.8 Gb/s) are transmitted from the FECs over fibers to a PCIe based readout unit. The FPGA engine on the readout unit compresses data for transmission to a server via 100 Gb ethernet. Components on the FECs are radiation tolerant. High data rates can be handled. The system is by design scalable and thus provides a functional prototype for high-rate streaming readout at Jefferson Lab and the future Electron Ion Collider. We have made fundamental measurements (noise, linearity, time resolution) on the SAMPA ASIC. We have also coupled the readout system to a small Gas Electron Multiplier (GEM) detector and have studied its response to cosmic rays. A beam test of the system is planned.

ABBOTT, David↗

Digital signal processor and processing method for GPS receivers

A digital signal processor and processing method therefor for use in receivers of the NAVSTAR/GLOBAL POSITIONING SYSTEM (GPS) employs a digital carrier down-converter, digital code correlator and digital tracking processor. The digital carrier down-converter and code correlator consists of an all-digital, minimum bit implementation that utilizes digital chip and phase advancers, providing exceptional control and accuracy in feedback phase and in feedback delay. Roundoff and commensurability errors can be reduced to extremely small values (e.g., less than 100 nanochips and 100 nanocycles roundoff errors and 0.1 millichip and 1 millicycle commensurability errors). The digital tracking processor bases the fast feedback for phase and for group delay in the C/A, P.sub.1, and P.sub.2 channels on the L.sub.1 C/A carrier phase thereby maintaining lock at lower signal-to-noise ratios, reducing errors in feedback delays, reducing the frequency of cycle slips and in some cases obviating the need for quadrature processing in the P channels. Simple and reliable methods are employed for data bit synchronization, data bit removal and cycle counting. Improved precision in averaged output delay values is provided by carrier-aided data-compression techniques. The signal processor employs purely digital operations in the sense that exactly the same carrier phase and group delay measurements are obtained, to the last decimal place, every time the same sampled data (i.e., exactly the same bits) are processed.

Thomas, Jr., Jess B.↗

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality—even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7 × faster than SZ3.

Wang, Daoce [University of Nebraska, Omaha]↗

Reconfigurable Hardware for Compressing Hyperspectral Image Data

High-speed, low-power, reconfigurable electronic hardware has been developed to implement ICER-3D, an algorithm for compressing hyperspectral-image data. The algorithm and parts thereof have been the topics of several NASA Tech Briefs articles, including Context Modeler for Wavelet Compression of Hyperspectral Images (NPO-43239) and ICER-3D Hyperspectral Image Compression Software (NPO-43238), which appear elsewhere in this issue of NASA Tech Briefs. As described in more detail in those articles, the algorithm includes three main subalgorithms: one for computing wavelet transforms, one for context modeling, and one for entropy encoding. For the purpose of designing the hardware, these subalgorithms are treated as modules to be implemented efficiently in field-programmable gate arrays (FPGAs). The design takes advantage of industry- standard, commercially available FPGAs. The implementation targets the Xilinx Virtex II pro architecture, which has embedded PowerPC processor cores with flexible on-chip bus architecture. It incorporates an efficient parallel and pipelined architecture to compress the three-dimensional image data. The design provides for internal buffering to minimize intensive input/output operations while making efficient use of offchip memory. The design is scalable in that the subalgorithms are implemented as independent hardware modules that can be combined in parallel to increase throughput. The on-chip processor manages the overall operation of the compression system, including execution of the top-level control functions as well as scheduling, initiating, and monitoring processes. The design prototype has been demonstrated to be capable of compressing hyperspectral data at a rate of 4.5 megasamples per second at a conservative clock frequency of 50 MHz, with a potential for substantially greater throughput at a higher clock frequency. The power consumption of the prototype is less than 6.5 W. The reconfigurability (by means of reprogramming) of the FPGAs makes it possible to effectively alter the design to some extent to satisfy different requirements without adding hardware. The implementation could be easily propagated to future FPGA generations and/or to custom application-specific integrated circuits.

Aranki, Nazeeh↗

Alternate physical formats for storing data in HDF

Since its inception HDF has evolved to meet new demands by the scientific community to support new kinds of data and data structures, larger data sets, and larger numbers of data sets. The first generation of HDF supported simple objects and simple storage schemes. These objects were used to build more complex objects such as raster images and scientific data sets. The second generation of HDF provided alternate methods of storing data elements, making it possible to do such things as store extendible objects with in HDF, to store data externally from HDF files, and support data compression effectively. As we look to the next generation of HDF, we are considering fundamental changes to HDF, including a redefinition of the basic HDF object from a simple object to a more general, higher-level scientific data object that has certain characteristics, such as dimensionality, a more general atomic number type, and attributes. These changes suggest corresponding changes to the HDF file format itself.

Folk, Mike↗

A comparison between ShapeFit compression and Full-Modelling method with PyBird for DESI 2024 and beyond

DESI aims to provide one of the tightest constraints on cosmological parameters by analysing the clustering of more than thirty million galaxies. However, obtaining such constraints requires special care in validating the methodology and efforts to reduce the computational time required through data compression and emulation techniques. In this work, we perform a rigorous validation of the PyBird power spectrum modelling code with both a traditional emulated Full-Modelling approach and the model-independent ShapeFit compression approach. By using cubic box simulations that accurately reproduce the clustering and precision of the DESI survey, we find that the cosmological constraints from ShapeFit and Full-Modelling are consistent with each other at the ∼ 0.5σ level for the ΛCDM model. Both ShapeFit and Full-Modelling are also consistent with the true ΛCDM simulation cosmology down to a scale of k max = 0.20 hMpc -1 even after including the hexadecapole. For extended models such as the wCDM and the oCDM models, we find that including the hexadecapole can significantly improve the constraints and reduce the modelling errors with the same k max . While their discrepancies between the constraints from ShapeFit and Full-Modelling are more significant than ΛCDM, they remain consistent within 0.7σ. Lastly, we also show that the constraints on cosmological parameters with the correlation function evaluated from PyBird down to s min = 30h -1 Mpc are unbiased and consistent with the constraints from the power spectrum.

79 ASTRONOMY AND ASTROPHYSICS↗

Data transmission networks

A task order was written by the High Resolution, High Frame Rate Video Technology (HHVT) project engineers to investigate data compression techniques that could be applied to the HHVT system, and both existing and planned downlink/uplink capabilities of the Space Shuttle and Space Station Freedom. The following tasks were included: (1) Investigate signal channel availability and determine both the maximum possible data rate and the average data rate; (2) Identify time blocks for HHVT video transmission assuming time sharing and interruptions in the communication links; (3) Determine the bit error rates to be expected; and (4) Define the transmit and receive interfaces. A summary chart of the data transmission capabilities for Tracking and Data Relay Satellite System (TDRSS), the Space Shuttle, Space Station Freedom, Spacelab, and USLab are also presented.

Alexovich, Robert↗

TRMM .25 deg x .25 deg Gridded Precipitation Text Product

Since the launch of the Tropical Rainfall Measuring Mission (TRMM), the Precipitation Measurement Missions science team has endeavored to provide TRMM precipitation retrievals in a variety of formats that are more easily usable by the broad science community than the standard Hierarchical Data Format (HDF) in which TRMM data is produced and archived. At the request of users, the Precipitation Processing System (PPS) has developed a .25 x .25 gridded product in an easily used ASCII text format. The entire TRMM mission data has been made available in this format. The paper provides the details of this new precipitation product that is designated with the TRMM designator 3G68.25. The format is packaged into daily files. It provides hourly precipitation information from the TRMM microwave imager (TMI), precipitation radar (PR), and TMI/PR combined rain retrievals. A major advantage of this approach is the inclusion only of rain data, compression when a particular grid has no rain from the PR or combined, and its direct ASCII text format. For those interested only in rain retrievals and whether rain is convection or stratiform, these products provide a huge reduction in the data volume inherent in the standard TRMM products. This paper provides examples of the 3G68 data products and their uses. It also provides information about C tools that can be used to aggregate daily files into larger time samples. In addition, it describes the possibilities inherent in the spatial sampling which allows resampling into coarser spatial sampling. The paper concludes with information about downloading the gridded text data products.

Stocker, Erich↗