Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 397 records · Page 22

Compressing Aviation Data in XML Format

Design, operations and maintenance activities in aviation involve analysis of variety of aviation data. This data is typically in disparate formats making it difficult to use with different software packages. Use of a self-describing and extensible standard called XML provides a solution to this interoperability problem. XML provides a standardized language for describing the contents of an information stream, performing the same kind of definitional role for Web content as a database schema performs for relational databases. XML data can be easily customized for display using Extensible Style Sheets (XSL). While self-describing nature of XML makes it easy to reuse, it also increases the size of data significantly. Therefore, transfemng a dataset in XML form can decrease throughput and increase data transfer time significantly. It also increases storage requirements significantly. A natural solution to the problem is to compress the data using suitable algorithm and transfer it in the compressed form. We found that XML-specific compressors such as Xmill and XMLPPM generally outperform traditional compressors. However, optimal use of Xmill requires of discovery of optimal options to use while running Xmill. This, in turn, depends on the nature of data used. Manual disc0ver.y of optimal setting can require an engineer to experiment for weeks. We have devised an XML compression advisory tool that can analyze sample data files and recommend what compression tool would work the best for this data and what are the optimal settings to be used with a XML compression tool.

Patel, Hemil↗

Transform coding for space applications

Data compression coding requirements for aerospace applications differ somewhat from the compression requirements for entertainment systems. On the one hand, entertainment applications are bit rate driven with the goal of getting the best quality possible with a given bandwidth. Science applications are quality driven with the goal of getting the lowest bit rate for a given level of reconstruction quality. In the past, the required quality level has been nothing less than perfect allowing only the use of lossless compression methods (if that). With the advent of better, faster, cheaper missions, an opportunity has arisen for lossy data compression methods to find a use in science applications as requirements for perfect quality reconstruction runs into cost constraints. This paper presents a review of the data compression problem from the space application perspective. Transform coding techniques are described and some simple, integer transforms are presented. The application of these transforms to space-based data compression problems is discussed. Integer transforms have an advantage over conventional transforms in computational complexity. Space applications are different from broadcast or entertainment in that it is desirable to have a simple encoder (in space) and tolerate a more complicated decoder (on the ground) rather than vice versa. Energy compaction with new transforms are compared with the Walsh-Hadamard (WHT), Discrete Cosine (DCT), and Integer Cosine (ICT) transforms.

Glover, Daniel↗

Screening IMS Seismic Detections using Dynamic Correlation Processor: Progress report for 2020

LLNL completed the FY20 workplan entailing application of the Dynamic Correlation Processor (DCP) to International Monitoring System (IMS) primary seismic data in 2019 and 2020. The effective analysis period was curtailed to January 1 through November 12, 2019 due to a data compression change that rendered later data unusable. Fixing the data compression error is a priority for our data source, and we plan to extend the period of data analysis when the fix is implemented. We found that the percentage overlap between detections generated by the International Data Centre (IDC) DFX software and detections generated by DCP varied widely by station. Overlap is 30-40% at some stations, suggesting that a large percentage of DFX detections could be screened (removed) from the detection-event association algorithm. However, detection overlap was much lower, a few percent, at many stations and the percentage overlap was low at stations known to have many mines and other local sources of seismicity. Unexpectedly low detection overlap at many stations prompted us to examine results for the ARCES station in detail. There is considerable mine activity near ARCES, but our initial FY20 analysis found only ~2% DCP/DFX detection overlap. Differences in the pre-processing (e.g. beam recipes) used by DCP and DFX appear to be the cause of low detection overlap at ARCES. DCP uses wideband filters to improve detector robustness and the IDC uses narrow-band pre-filtering to improve sensitivity. After reprocessing ARCES using the IDC beam recipe, DCP/DFX overlap increased to ~74%. This result shows that we must reprocess the IMS network using the IDC station-specific beam recipes if we are to effectively screen DFX detections and improve automatic event building performance at the IDC.

58 GEOSCIENCES↗

Subband/transform functions for image processing

Functions for image data processing written for use with the MATLAB(TM) software package are presented. These functions provide the capability to transform image data with block transformations (such as the Walsh Hadamard) and to produce spatial frequency subbands of the transformed data. Block transforms are equivalent to simple subband systems. The transform coefficients are reordered using a simple permutation to give subbands. The low frequency subband is a low resolution version of the original image, while the higher frequency subbands contain edge information. The transform functions can be cascaded to provide further decomposition into more subbands. If the cascade is applied to all four of the first stage subbands (in the case of a four band decomposition), then a uniform structure of sixteen bands is obtained. If the cascade is applied only to the low frequency subband, an octave structure of seven bands results. Functions for the inverse transforms are also given. These functions can be used for image data compression systems. The transforms do not in themselves produce data compression, but prepare the data for quantization and compression. Sample quantization functions for subbands are also given. A typical compression approach is to subband the image data, quantize it, then use statistical coding (e.g., run-length coding followed by Huffman coding) for compression. Contour plots of image data and subbanded data are shown.

Glover, Daniel↗

Image Compression Algorithm Altered to Improve Stereo Ranging

A report discusses a modification of the ICER image-data-compression algorithm to increase the accuracy of ranging computations performed on compressed stereoscopic image pairs captured by cameras aboard the Mars Exploration Rovers. (ICER and variants thereof were discussed in several prior NASA Tech Briefs articles.) Like many image compressors, ICER was designed to minimize a mean-square-error measure of distortion in reconstructed images as a function of the compressed data volume. The present modification of ICER was preceded by formulation of an alternative error measure, an image-quality metric that focuses on stereoscopic-ranging quality and takes account of image-processing steps in the stereoscopic-ranging process. This metric was used in empirical evaluation of bit planes of wavelet-transform subbands that are generated in ICER. The present modification, which is a change in a bit-plane prioritization rule in ICER, was adopted on the basis of this evaluation. This modification changes the order in which image data are encoded, such that when ICER is used for lossy compression, better stereoscopic-ranging results are obtained as a function of the compressed data volume.

Kiely, Aaron↗

Total variation vs L1 regularization: a comparison of compressive sensing optimization methods for chemical detection

One of the fundamental assumptions of compressive sensing (CS) is that a signal can be reconstructed from a small number of samples by solving an optimization problem with the appropriate regularization term. Two standard regularization terms are the L1 norm and the total variation (TV) norm. We present a comparison of CS reconstruction results based on these two approaches in the context of chemical detection, and we demonstrate that optimization based on the L1 norm outperforms optimization based on the TV norm. Our comparison is driven by CS sampling, reconstruction, and chemical detection in two real-world datasets: the Physical Sciences Inc. Fabry-Perot interferometer sensor multispectral dataset and the Johns Hopkins Applied Physics Lab FTIR-based longwave infrared sensor hyperspectral dataset. Both datasets contain the release of a chemical simulant such as glacial acetic acid, triethyl phosphate, and sulfur hexafluoride. For chemical detection we use the adaptive coherence estimator (ACE) and bulk coherence, and we propose algorithmic ACE thresholds to define the presence or absence of a chemical of interest in both un-compressed data cubes and reconstructed data cubes. The un-compressed data cubes provide an approximate ground truth. We demonstrate that optimization based on either the L1 norm or TV norm results in successful chemical detection at a compression rate of 90%, but we show that L1 optimization is preferable. We present quantitative comparisons of chemical detection on reconstructions from the two methods, with an emphasis on the number of pixels with an ACE value above the threshold.

Hyperspectral imagery, L1-norm, total variation, A↗

Effects of compression and random noise on multispectral data.

Two problem areas are often encountered in the use of multispectral scanner data: (1) large quantities of man and machine time are required to analyze and store the volumes of data gathered and (2) the quality of the data is usually reduced by the introduction of unwanted random noise. This paper will present some effects of data compression and random noise on multispectral data, particularly as they apply to pattern recognition and picture quality.

Ready, P. J.↗

The Digital Revolution in Measurements

This paper considers what it means to make a measurement, and the changes in measurement technology over the years. The impact of the latest changes, which have resulted in most electrical measurements being done digitally, is explored. It is argued that the process of measurement can be considered equivalent to one of data compression. The smart grid will certainly result in many more signals being made available, and therefore a great deal of data compression will be taking place. Measurements will be made in parts of the power system presently unmonitored, as well as parts that are already well covered by instrumentation. The smart grid engineer must decide what it means to have “useful” information. Unless care is taken, the signal processing may furnish information that is not useful, and may not even make sense. The paper concludes by examining the possibilities of data compression from multiple separate signals.

abstraction↗

Real-time and post-hoc compression for data from Distributed Acoustic Sensing

Distributed Acoustic Sensing (DAS) is an emerging sensing technology that records the strain-rate along fiber optic cables at high spatial and temporal resolution. This technique is becoming a popular tool in seismology, hydrology, and other subsurface monitoring applications. However, due to the large coverage (10’s of km) and high density of measurements (1m spacing at 100’s of Hz), a DAS installation could produce terabytes of data records per day. Because many DAS instruments are deployed in remote locations, this large data size poses significant challenges to its transfer and storage. In this paper, we explore lossless compression methods to reduce the storage requirement in both real-time and post-hoc scenarios. Here we propose a two-stage compression method to improve the compression ratio and compression speed. This two-stage compression method could reduce the storage requirement by 40%, which is 20% more than other lossless methods, such as ZSTD. We demonstrate that the compression method could complete its operation well before the DAS instrument needs to output the next file, making it suitable for real-time DAS acquisition. We also implement a parallel compression method for a post-hoc scenario and demonstrate that our method could effectively utilize a parallel computer. With 256 CPU cores, our parallel compression method achieves the speed of 26GB/second.

58 GEOSCIENCES↗

A VLSI design for universal noiseless coding

The practical, noiseless and efficient data-compression technique presented involves a conceptual VLSI design which is capable of meeting real-time processing rates and meets low-power, low-weight, and small-volume requirements. This form of data compression is applicable to image data compression aboard future low-budget spaceflight missions, for such instruments as visual-IR mapping spectrometers and high-resolution imaging spectrometers.

Lee, Jun-Ji↗

Bounding Methods for Heterogeneous Lidar-derived Navigational Geofences

Safe Unmanned Aerial Vehicle (UAV) operations near the ground require navigation methods that avoid fixed obstacles such as buildings, power lines and trees. Aerial lidar surveys of ground structures are available with the precision and accuracy to geolocate obstacles, but the high volume of raw survey data can exceed the compute power of onboard processors and the rendering ability of ground-based flight planning maps. Representing ground structures with bounding polyhedra instead of point clouds greatly reduces the data size and can enable effective obstacle avoidance, as long as the bounding geometry envelopes the structures with high spatial fidelity. This report describes in detail four methods to compute bounding geometries of ground obstacles from lidar point clouds. The four methods are: 1) 2.5D Maximum Elevation Box, 2) 2.5D Ground Map Extrusion, 3) 3D Bounding Cylinder, and 4) 3D Bounding Box. The methods are applied to five point cloud datasets from lidar surveys of UAV flight research sites in Georgia and Virginia with an average point spacing that ranges from 0.1m to 0.6m. The methods are assessed using survey areas with geometrically heterogeneous ground structures: buildings, vegetation, power lines, and sub-meter structures such as road signs and guy wires. The 2.5D Maximum Elevation Box method is useful for simple structures. The 2.5D Ground Map Extrusion method efficiently encloses vegetation, but requires handdrawn ground footprints. The 3D Bounding Cylinder method excels at enclosing linear structures such as power lines and fences. The 3D Bounding Box method excels at enclosing planar structures such as buildings. The methods are compared on the basis of data compression and boundary fidelity on selected areas. The 2.5D methods yield the highest data compression but the polyhedra produced by them enclose significant amounts of empty space. Boundary fidelity is superior for the 3D methods, though this fidelity comes at the cost of a roughly thirtyfold lower data compression ratio than the 2.5D Maximum Elevation Box method. A mix of these output geometries is proposed for autonomous UAV navigation with limited on-board computing. Both the accuracy and spatial detail of emerging satellite-based survey technology lower than that of aerial lidar scanning survey technology. Sub-meter structures and thin linear structures are not reliably mapped at present by satellite-based surveys.

Moore, Andrew J.↗

Inverse Method for Estimation of Composite Kink-Band Toughness from Open-Hole Compression Strength Data

Fiber-reinforced polymer matrix composite materials can fail by kink-band propagation mechanism when subjected to in-plane compressive loading. This mode of failure is especially prevalent in compressive loading of laminates with holes, cut-outs, or impact damage. Most of the successful models for predicting compressive strength of such laminates require “fracture” toughness associated with kink-band propagation under in-plane compression. However, this property is difficult to measure experimentally, limiting the use of such models in design practice. In this paper an inverse method is proposed to estimate the kink-band toughness of the laminate from its open-hole compression strength data, which is an easier property to measure experimentally. Furthermore, a scaling relationship is proposed to estimate kink-band toughness for other laminate configurations of the same material.

Luke Borkowski↗

Real-Time SCADA Cyber Protection Using Compression Techniques

The Department of Energy’s Office of Electricity Delivery and Energy Reliability (DOE-OE) has a critical mission to secure the energy infrastructure from cyber attack. Through DOE-OE’s Cybersecurity for Energy Delivery Systems (CEDS) program, the Idaho National Laboratory (INL) has developed a method to detect malicious traffic on Supervisory, Control, and Data Acquisition (SCADA) network using a data compression technique. SCADA network traffic is often repetitive with only minor differences between packets. Research performed at the INL showed that SCADA network traffic has traits desirable for using compression analysis to identify abnormal network traffic. An open source implementation of a Lempel-Ziv-Welch (LZW) lossless data compression algorithm was used to compress and analyze surrogate SCADA traffic. Infected SCADA traffic was found to have statistically significant differences in compression when compared against normal SCADA traffic at the packet level. The initial analyses and results are clearly able to identify malicious network traffic from normal traffic at the packet level with a very high confidence level across multiple ports and traffic streams. Statistical differentiation between infected and normal traffic level was possible using a modified data compression technique at the 99% probability level for all data analyzed. However, the conditions tested were rather limited in scope and need to be expanded into more realistic simulations of hacking events using techniques and approaches that are better representative of a real-world attack on a SCADA system. Nonetheless, the use of compression techniques to identify malicious traffic on SCADA networks in real time appears to have significant merit for infrastructure protection.

97 MATHEMATICS AND COMPUTING↗

Accelerating Lossy and Lossless Compression on Emerging BlueField DPU Architectures

Data compression has become a crucial technique in addressing performance bottlenecks caused by increasing data volumes in High-Performance Computing (HPC), Big Data, and Deep Learning (DL). Despite its potential to boost system performance, recent studies have identified significant challenges with existing compression methods, mainly due to their high computational demands amidst continuously growing data sizes. Concurrently, the advent of Data Processing Units (DPUs), equipped with programmable System-on-Chip (SoC) and specialized compression accelerators, offers a promising opportunity to alter the landscape of data compression. This paper explores the complexities and potential of leveraging NVIDIA BlueField DPUs to accelerate lossy and lossless compression. Towards this, we introduce PEDAL, an innovative library that leverages the hardware capabilities of DPUs to unify and optimize data compression designs. Moreover, we seamlessly co-design PEDAL with the popular MPICH MPI library, demonstrating up to 101x speedup in compression time and 88x decrease in communication latency. Drawing on these achievements, we share our experience with various research communities about accelerating data compression on DPUs in communication-oriented HPC scenarios.

Li, Yuke↗

Analysis Resistant Cipher Method and Apparatus

A system for encoding and decoding data words including an anti-analysis encoder unit for receiving an original plaintext and producing a recoded data, a data compression unit for receiving the recoded data and producing a compressed recoded data, and an encryption unit for receiving the compressed recoded data and producing an encrypted data. The recoded data has an increased non-correlatable data redundancy compared with the original plaintext in order to mask the statistical distribution of characters in the plaintext data. The system of the present invention further includes a decryption unit for receiving the encrypted data and producing a decrypted data, a data decompression unit for receiving the decrypted data and producing an uncompressed recoded data, and an anti-analysis decoder unit for receiving the uncompressed recoded data and producing a recovered plaintext that corresponds with the original plaintext.

Oakley, Ernest C.↗