Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 127 records · Page 7

Distributed video coding for arrays of remote sensing nodes : final report.

This document is the final report for the Sandia National Laboratory funded Student Fellowship position at New Mexico State University (NMSU) from 2008 to 2010. Ivan Mecimore, the PhD student in Electrical Engineering at NMSU, was conducting research into image and video processing techniques to identify features and correlations within images without requiring the decoding of the data compression. Such an analysis technique would operate on the encoded bit stream, potentially saving considerable processing time when operating on a platform that has limited computational resources. Unfortunately, the student has elected in mid-year not to continue with his research or the fellowship position. The student is unavailable to provide any details of his research for inclusion in this final report. As such, this final report serves solely to document the information provided in the previous end of year summary.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Active Flash: Performance-Energy Tradeoffs for Out-of-Core Processing on Non-Volatile Memory Devices

In this abstract, we study the performance and energy tradeoffs involved in migrating data analysis into the flash device, a process we refer to as Active Flash. The Active Flash paradigm is similar to 'active disks', which has received considerable attention. Active Flash allows us to move processing closer to data, thereby minimizing data movement costs and reducing power consumption. It enables true out-of-core computation. The conventional definition of out-of-core solvers refers to an approach to process data that is too large to fit in the main memory and, consequently, requires access to disk. However, in Active Flash, processing outside the host CPU literally frees the core and achieves real 'out-of-core' analysis. Moving analysis to data has long been desirable, not just at this level, but at all levels of the system hierarchy. However, this requires a detailed study on the tradeoffs involved in achieving analysis turnaround under an acceptable energy envelope. To this end, we first need to evaluate if there is enough computing power on the flash device to warrant such an exploration. Flash processors require decent computing power to run the internal logic pertaining to the Flash Translation Layer (FTL), which is responsible for operations such as address translation, garbage collection (GC) and wear-leveling. Modern SSDs are composed of multiple packages and several flash chips within a package. The packages are connected using multiple I/O channels to offer high I/O bandwidth. SSD computing power is also expected to be high enough to exploit such inherent internal parallelism within the drive to increase the bandwidth and to handle fast I/O requests. More recently, SSD devices are being equipped with powerful processing units and are even embedded with multicore CPUs (e.g. ARM Cortex-A9 embedded processor is advertised to reach 2GHz frequency and deliver 5000 DMIPS; OCZ RevoDrive X2 SSD has 4 SandForce controllers, each with 780MHz max frequency Tensilica core). Efforts that take advantage of the available computing cycles on the processors on SSDs to run auxiliary tasks other than actual I/O requests are beginning to emerge. Kim et al. investigate database scan operations in the context of processing on the SSDs, and propose dedicated hardware logic to speed up scans. Also, cluster architectures have been explored, which consist of low-power embedded CPUs coupled with small local flash to achieve fast, parallel access to data. Processor utilization on SSD is highly dependent on workloads and, therefore, they can be idle during periods with no I/O accesses. We propose to use the available processing capability on the SSD to run tasks that can be offloaded from the host. This paper makes the following contributions: (1) We have investigated Active Flash and its potential to optimize the total energy cost, including power consumption on the host and the flash device; (2) We have developed analytical models to analyze the performance-energy tradeoffs for Active Flash, by treating the SSD as a blackbox, this is particularly valuable due to the proprietary nature of the SSD internal hardware; and (3) We have enhanced a well-known SSD simulator (from MSR) to implement 'on-the-fly' data compression using Active Flash. Our results provide a window into striking a balance between energy consumption and application performance.

42 ENGINEERING↗

Screening IMS Seismic Detections using Dynamic Correlation Processor: Progress report for 2020

LLNL completed the FY20 workplan entailing application of the Dynamic Correlation Processor (DCP) to International Monitoring System (IMS) primary seismic data in 2019 and 2020. The effective analysis period was curtailed to January 1 through November 12, 2019 due to a data compression change that rendered later data unusable. Fixing the data compression error is a priority for our data source, and we plan to extend the period of data analysis when the fix is implemented. We found that the percentage overlap between detections generated by the International Data Centre (IDC) DFX software and detections generated by DCP varied widely by station. Overlap is 30-40% at some stations, suggesting that a large percentage of DFX detections could be screened (removed) from the detection-event association algorithm. However, detection overlap was much lower, a few percent, at many stations and the percentage overlap was low at stations known to have many mines and other local sources of seismicity. Unexpectedly low detection overlap at many stations prompted us to examine results for the ARCES station in detail. There is considerable mine activity near ARCES, but our initial FY20 analysis found only ~2% DCP/DFX detection overlap. Differences in the pre-processing (e.g. beam recipes) used by DCP and DFX appear to be the cause of low detection overlap at ARCES. DCP uses wideband filters to improve detector robustness and the IDC uses narrow-band pre-filtering to improve sensitivity. After reprocessing ARCES using the IDC beam recipe, DCP/DFX overlap increased to ~74%. This result shows that we must reprocess the IMS network using the IDC station-specific beam recipes if we are to effectively screen DFX detections and improve automatic event building performance at the IDC.

58 GEOSCIENCES↗

Total variation vs L1 regularization: a comparison of compressive sensing optimization methods for chemical detection

One of the fundamental assumptions of compressive sensing (CS) is that a signal can be reconstructed from a small number of samples by solving an optimization problem with the appropriate regularization term. Two standard regularization terms are the L1 norm and the total variation (TV) norm. We present a comparison of CS reconstruction results based on these two approaches in the context of chemical detection, and we demonstrate that optimization based on the L1 norm outperforms optimization based on the TV norm. Our comparison is driven by CS sampling, reconstruction, and chemical detection in two real-world datasets: the Physical Sciences Inc. Fabry-Perot interferometer sensor multispectral dataset and the Johns Hopkins Applied Physics Lab FTIR-based longwave infrared sensor hyperspectral dataset. Both datasets contain the release of a chemical simulant such as glacial acetic acid, triethyl phosphate, and sulfur hexafluoride. For chemical detection we use the adaptive coherence estimator (ACE) and bulk coherence, and we propose algorithmic ACE thresholds to define the presence or absence of a chemical of interest in both un-compressed data cubes and reconstructed data cubes. The un-compressed data cubes provide an approximate ground truth. We demonstrate that optimization based on either the L1 norm or TV norm results in successful chemical detection at a compression rate of 90%, but we show that L1 optimization is preferable. We present quantitative comparisons of chemical detection on reconstructions from the two methods, with an emphasis on the number of pixels with an ACE value above the threshold.

Hyperspectral imagery, L1-norm, total variation, A↗

The Digital Revolution in Measurements

This paper considers what it means to make a measurement, and the changes in measurement technology over the years. The impact of the latest changes, which have resulted in most electrical measurements being done digitally, is explored. It is argued that the process of measurement can be considered equivalent to one of data compression. The smart grid will certainly result in many more signals being made available, and therefore a great deal of data compression will be taking place. Measurements will be made in parts of the power system presently unmonitored, as well as parts that are already well covered by instrumentation. The smart grid engineer must decide what it means to have “useful” information. Unless care is taken, the signal processing may furnish information that is not useful, and may not even make sense. The paper concludes by examining the possibilities of data compression from multiple separate signals.

abstraction↗

Real-time and post-hoc compression for data from Distributed Acoustic Sensing

Distributed Acoustic Sensing (DAS) is an emerging sensing technology that records the strain-rate along fiber optic cables at high spatial and temporal resolution. This technique is becoming a popular tool in seismology, hydrology, and other subsurface monitoring applications. However, due to the large coverage (10’s of km) and high density of measurements (1m spacing at 100’s of Hz), a DAS installation could produce terabytes of data records per day. Because many DAS instruments are deployed in remote locations, this large data size poses significant challenges to its transfer and storage. In this paper, we explore lossless compression methods to reduce the storage requirement in both real-time and post-hoc scenarios. Here we propose a two-stage compression method to improve the compression ratio and compression speed. This two-stage compression method could reduce the storage requirement by 40%, which is 20% more than other lossless methods, such as ZSTD. We demonstrate that the compression method could complete its operation well before the DAS instrument needs to output the next file, making it suitable for real-time DAS acquisition. We also implement a parallel compression method for a post-hoc scenario and demonstrate that our method could effectively utilize a parallel computer. With 256 CPU cores, our parallel compression method achieves the speed of 26GB/second.

58 GEOSCIENCES↗

Real-Time SCADA Cyber Protection Using Compression Techniques

The Department of Energy’s Office of Electricity Delivery and Energy Reliability (DOE-OE) has a critical mission to secure the energy infrastructure from cyber attack. Through DOE-OE’s Cybersecurity for Energy Delivery Systems (CEDS) program, the Idaho National Laboratory (INL) has developed a method to detect malicious traffic on Supervisory, Control, and Data Acquisition (SCADA) network using a data compression technique. SCADA network traffic is often repetitive with only minor differences between packets. Research performed at the INL showed that SCADA network traffic has traits desirable for using compression analysis to identify abnormal network traffic. An open source implementation of a Lempel-Ziv-Welch (LZW) lossless data compression algorithm was used to compress and analyze surrogate SCADA traffic. Infected SCADA traffic was found to have statistically significant differences in compression when compared against normal SCADA traffic at the packet level. The initial analyses and results are clearly able to identify malicious network traffic from normal traffic at the packet level with a very high confidence level across multiple ports and traffic streams. Statistical differentiation between infected and normal traffic level was possible using a modified data compression technique at the 99% probability level for all data analyzed. However, the conditions tested were rather limited in scope and need to be expanded into more realistic simulations of hacking events using techniques and approaches that are better representative of a real-world attack on a SCADA system. Nonetheless, the use of compression techniques to identify malicious traffic on SCADA networks in real time appears to have significant merit for infrastructure protection.

97 MATHEMATICS AND COMPUTING↗

Accelerating Lossy and Lossless Compression on Emerging BlueField DPU Architectures

Data compression has become a crucial technique in addressing performance bottlenecks caused by increasing data volumes in High-Performance Computing (HPC), Big Data, and Deep Learning (DL). Despite its potential to boost system performance, recent studies have identified significant challenges with existing compression methods, mainly due to their high computational demands amidst continuously growing data sizes. Concurrently, the advent of Data Processing Units (DPUs), equipped with programmable System-on-Chip (SoC) and specialized compression accelerators, offers a promising opportunity to alter the landscape of data compression. This paper explores the complexities and potential of leveraging NVIDIA BlueField DPUs to accelerate lossy and lossless compression. Towards this, we introduce PEDAL, an innovative library that leverages the hardware capabilities of DPUs to unify and optimize data compression designs. Moreover, we seamlessly co-design PEDAL with the popular MPICH MPI library, demonstrating up to 101x speedup in compression time and 88x decrease in communication latency. Drawing on these achievements, we share our experience with various research communities about accelerating data compression on DPUs in communication-oriented HPC scenarios.

Li, Yuke↗

Less is More: Bigger Data from Compressive Measurements

Compressive sensing approaches are beginning to take hold in (scanning) transmission electron microscopy (S/TEM) [1,2,3]. Compressive sensing is a mathematical theory about acquiring signals in a compressed form (measurements) and the probability of recovering the original signal by solving an inverse problem [4]. The inverse problem is underdetermined (more unknowns than measurements), so it is not obvious that recovery is possible. Compression is achieved by taking inner products of the signal with measurement weight vectors. Both Gaussian random weights and Bernoulli (0,1) random weights form a large class of measurement vectors for which recovery is possible. The measurements can also be designed through an optimization process. The key insight for electron microscopists is that compressive sensing can be used to increase acquisition speed and reduce dose. Building on work initially developed for optical cameras, this new paradigm will allow electron microscopists to solve more problems in the engineering and life sciences. We will be collecting orders of magnitude more data than previously possible. The reason that we will have more data is because we will have increased temporal/spatial/spectral sampling rates, and we will be able ability to interrogate larger classes of samples that were previously too beam sensitive to survive the experiment. For example consider an in-situ experiment that takes 1 minute. With traditional sensing, we might collect 5 images per second for a total of 300 images. With compressive sensing, each of those 300 images can be expanded into 10 more images, making the collection rate 50 images per second, and the decompressed data a total of 3000 images [3]. But, what are the implications, in terms of data, for this new methodology? Acquisition of compressed data will require downstream reconstruction to be useful. The reconstructed data will be much larger than traditional data, we will need space to store the reconstructions during analysis, and the computational demands for analysis will be higher. Moreover, there will be time costs associated with reconstruction. Deep learning [5] is an approach to address these problems. Deep learning is a hierarchical approach to find useful (for a particular task) representations of data. Each layer of the hierarchy is intended to represent higher levels of abstraction. For example, a deep model of faces might have sinusoids, edges and gradients in the first layer; eyes, noses, and mouths in the second layer, and faces in the third layer. There has been significant effort recently in deep learning algorithms for tasks beyond image classification such as compressive reconstruction [6] and image segmentation [7]. A drawback of deep learning, however, is that training the model requires large datasets and dedicated computational resources (to reduce training time to a few days). A second issue is that deep learning is not user-friendly and the meaning behind the results is usually not interpretable. We have shown it is possible to reduce the data set size while maintaining model quality [8] and developed interpretable models for image classification [9], but the demands are still significant. The key to addressing these problems is to NOT reconstruct the data. Instead, we should design computational sensors that give answers to specific problems. A simple version of this idea is compressive classification [10], where the goal is to classify signal type from a small number of compressed measurements. Classification is a much simpler problem than reconstruction, so 1) much fewer measurements will be necessary, and 2) these measurements will probably not be useful for reconstruction. Other simple examples of computational sensing include determining object volume or the number of objects present in the field of view [11].

Stevens, Andrew↗

Time-temperature-stress superposition of PBX 9502 compressive creep data for lifetime predictions

The plastic-bonded explosive (PBX) 9502 is a viscoelastic, high-solids loaded polymer bound composite comprised of 95 weight% (wt%) TATB explosive crystals and 5 wt% FK-800 polymer binder. The uniaxial quasi-static mechanical properties have been studied and characterized extensively over 25+ years, including creep. Creep is the strain evolution over time as a result of an asymmetric load typically at temperatures above ambient. Creep measurements to failure can take excessively long periods of time depending on various factors including applied stress, temperature, and loading rate that render a single test to failure impossible. Using the principles of time-temperature-stress superposition (TTSSP), short-term creep tests at various applied stresses and temperatures can be used for long-term creep predictions. The TTSSP model shifts short-term test data using two factors, one for temperature and stress, creating a master curve at a reference test condition. The master curve can be shifted to predict the creep response at any desired test condition. PBX 9502 compression specimens were tested at three temperatures and various applied stresses. This work discusses the TTSSP analysis of PBX 9502 compressive creep data to create a single comprehensive master curve.

36 MATERIALS SCIENCE↗

Time-Temperature-Stress Superposition of PBX 9502 Compressive Creep Data for Lifetime Predictions

The plastic-bonded explosive (PBX) 9502 is a viscoelastic, high-solids loaded polymer bound composite comprised of 95 weight% (wt%) TATB explosive crystals and 5 wt% FK-800 polymer binder. The uniaxial quasi-static mechanical properties have been studied and characterized extensively over 25+ years, including creep. Creep is the strain evolution over time as a result of an asymmetric load typically at temperatures above ambient. Creep measurements to failure can take excessively long periods of time depending on various factors including applied stress, temperature, and loading rate that render a single test to failure impossible. Using the principles of time-temperature-stress superposition (TTSSP), short-term creep tests at various applied stresses and temperatures can be used for long-term creep predictions. The TTSSP model shifts short-term test data using two factors, one for temperature and stress, creating a master curve at a reference test condition. The master curve can be shifted to predict the creep response at any desired test condition. PBX 9502 compression specimens were tested at three temperatures and various applied stresses. This work discusses the TTSSP analysis of PBX 9502 compressive creep data to create a single comprehensive master curve.

36 MATERIALS SCIENCE↗

A Novel Event Detection Method Using PMU Data with High Precision

To take full advantage of the considerably high reporting rate of phasor measurement units (PMU) data, this paper develops a novel PMU-based event detection methodology. Considering the huge amount of streaming PMU data, a data compression algorithm, swinging door trending (SDT), is first used to compress the PMU data and generate multiple compression intervals. Then dynamic programming is utilized to solve the optimization problem, which is recursively constituted by a score function. Based on predefined PMU event rules, dynamic programming merges adjacent compression intervals with the same slope direction. Finally, all the PMU event features are characterized. A conventional wavelet-based event detection method is compared with the developed dynamic programming based SDT (DPSDT) method. Numerical simulations on the real-time and synthetic PMU data show that the DPSDT method can accurately detect the start-time of an event and the event placement with relatively high precision. Furthermore, the PMU event features, including the magnitude and duration of strokes, are characterized.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Streaming Compression of Scientific Data via Weak-SINDy

Here, in this paper, a streaming weak-SINDy algorithm is developed specifically for compressing streaming scientific data. The production of scientific data, either via simulation or experiments, is undergoing a stage of exponential growth, which makes data compression important and often necessary for storing and utilizing large scientific data sets. As opposed to classical “offline” compression algorithms that perform compression on a readily available data set, streaming compression algorithms compress data “online” while the data generated from simulation or experiments is still flowing through the system. This feature makes streaming compression algorithms well suited for scientific data compression, where storing the full data set offline is often infeasible. This work proposes a new streaming compression algorithm, streaming weak-SINDy, which takes advantage of the underlying data characteristics during compression. The streaming weak-SINDy algorithm constructs feature matrices and target vectors in the online stage via a streaming integration method in a memory efficient manner. The feature matrices and target vectors are then used in the offline stage to build a model through a regression process that aims to recover equations that govern the evolution of the data. For compressing high-dimensional streaming data, we adopt a streaming proper orthogonal decomposition (POD) process to reduce the data dimension and then use the streaming weak-SINDy algorithm to compress the temporal data of the POD expansion. We propose modifications to the streaming weak-SINDy algorithm to accommodate the dynamically updated POD basis. By combining the built model from the streaming weak-SINDy algorithm and a small amount of data samples, the full data flow could be reconstructed accurately at a low memory cost, as shown in the numerical tests.

97 MATHEMATICS AND COMPUTING↗

Prior data assisted compressed sensing: A novel MR imaging strategy for real time tracking of lung tumors

Purpose: Hybrid radiotherapy-MRI devices promise real time tracking of moving tumors to focus the radiation portals to the tumor during irradiation. This approach will benefit from the increased temporal resolution of MRI's data acquisition and reconstruction. In this work, the authors propose a novel spatial-temporal compressed sensing (CS) imaging strategy for the real time MRI–-prior data assisted compressed sensing (PDACS), which aims to improve the image quality of the conventional CS without significantly increasing reconstruction times. Methods: Conventional 2D CS requires a random sampling of partial k-space data, as well as an iterative reconstruction that simultaneously enforces the image's sparsity in a transform domain as well as maintains the fidelity to the acquired k-space. PDACS method requires the additional acquisition of the prior data, and for reconstruction, it additionally enforces fidelity to the prior k-space domain similar to viewsharing. In this work, the authors evaluated the proposed PDACS method by comparing its results to those obtained from the 2D CS and viewsharing methods when performed individually. All three methods are used to reconstruct images from lung cancer patients whose tumors move and who are likely to benefit from lung tumor tracking. The patients are scanned, using a 3T MRI, under free breathing using the fully sampled k-space with 2D dynamic bSSFP sequence in a sagittal plane containing lung tumor. These images form a reference set for the evaluation of the partial k-space methods. To create partial k-space, the fully sampled k-space is retrospectively undersampled to obtain a range of acquisition acceleration factors, and reconstructed with 2D-CS, PDACS, and viewshare methods. For evaluation, metrics assessing global image artifacts as well as tumor contour shape fidelity are determined from the reconstructed images. These analyses are performed both for the original 3T images and those at a simulated 0.5T equivalent noise level. Results: In the 3.0T images, the PDACS strategy is shown to give superior results compared to viewshare and conventional 2D CS using all metrics. The 2D-CS tends to perform better than viewshare at the low acceleration factors, while the opposite is true at the high acceleration factors. At simulated 0.5T images, PDACS method performs only marginally better than the viewsharing method, both of which are superior compared to 2D CS. The PDACS image reconstruction time (0.3 s/image) is similar to that of the conventional 2D CS. Conclusions: The PDACS method can potentially improve the real time tracking of moving tumors by significantly increasing MRI's data acquisition speeds. In 3T images, the PDACS method does provide a benefit over the other two methods in terms of both the overall image quality and the ability to accurately and automatically contour the tumor shape. MRI's data acquisition may be accelerated using the simpler viewsharing strategy at the lower, 0.5T magnetic field, as the marginal benefit of the PDACS method may not justify its additional reconstruction times.

60 APPLIED LIFE SCIENCES↗

Adaptive wavelet compression of large additive manufacturing experimental and simulation datasets

New manufacturing technologies such as additive manufacturing require research and development to minimize the uncertainties in the produced parts. The research involves experimental measurements and large simulations, which result in huge quantities of data to store and analyze. We address this challenge by alleviating the data storage requirements using lossy data compression. We select wavelet bases as the mathematical tool for compression. Unlike images, additive manufacturing data is often represented on irregular geometries and unstructured meshes. Thus, we use Alpert tree-wavelets as bases for our data compression method. We first analyze different basis functions for the wavelets and find the one that results in maximal compression and miminal error in the reconstructed data. We then devise a new adaptive thresholding method that is data-agnostic and allows a priori estimation of the reconstruction error. Finally, we propose metrics to quantify the global and local errors in the reconstructed data. One of the error metrics addresses the preservation of physical constraints in reconstructed data fields, such as divergence-free stress field in structural simulations. While our compression and decompression method is general, we apply it to both experimental and computational data obtained from measurements and thermal/structural modeling of the sintering of a hollow cylinder from metal powders using a Laser Engineered Net Shape process. The results show that monomials achieve optimal compression performance when used as wavelet bases. The new thresholding method results in compression ratios that are two to seven times larger than the ones obtained with commonly used thresholds. Overall, adaptive Alpert tree-wavelets can achieve compression ratios between one and three orders of magnitude depending on the features in the data that are required to preserve. Furthermore, these results show that Alpert tree-wavelet compression is a viable and promising technique to reduce the size of large data structures found in both experiments and simulations.

42 ENGINEERING↗

The Exploitation of Data Reduction for Visualization

The disparity between the computational speed and storage bandwidth, as demonstrated in Figure 1, is a well known problem that grows with each successive generation. The visualization community is principally responding to this issue by using in situ to reduce which data must be written to storage. However, other communities are taking different, possibly complementary approaches. In particular, data compression is a common general approach to reduce storage demands. Data compression technologies are typically not designed with post processing in mind. The principal metrics measured are compression ratio, the improved bandwidth to storage, and the error introduced. It is assumed that data is inflated to its full size before any post processing can happen. Although when talking about bandwidth disparities, HPC’s dirty little secret is that no part of the memory nor interconnect hardware is increasing at the rate of computation. For example, the Summit supercomputer has a peak computation rate almost 10 times its predecessor, Titan, but only about 4 times the memory, less than twice the aggregate memory bandwidth, and almost no improvement in the interconnect bisection bandwidth. Naively inflating data for post processing does not help with limitations in the memory and interconnect systems.

97 MATHEMATICS AND COMPUTING↗

Noise and Bias in Square-Root Compression Schemes

We investigate data compression schemes for proposed all-sky diffraction-limited visible/NIR sky surveys aimed at the dark energy problem. We show that lossy square-root compression to 1 bit of noise per pixel, followed by standard lossless compression algorithms, reduces the images to 2.5-4 bits per pixel, depending primarily upon the level of cosmic-ray contamination of the images. Compression to this level adds noise equivalent to a 10% penalty in observing time. We derive an analytic correction to flux biases inherent to the square-root compression scheme. Numerical tests on simple galaxy models confirm that galaxy fluxes and shapes are measured with systematic biases <~10^{-4} induced by the compression scheme, well below the requirements of supernova and weak gravitational lensing dark-energy experiments. An accompanying paper (Vanderveld 2009) bounds the shape biases using realistic simulated images of the high-Galactic-latitude sky.

Bernstein, Gary M.↗

Malicious activity detection in a memory

A method and apparatus for monitoring a volatile memory in a computer system. Samples of compressed data from locations in the volatile memory in the computer system are read. Data in the volatile memory is reconstructed using the samples of compressed data. The data is an image of the volatile memory. The image enables determining whether an undesired process is present in the volatile memory.

Wheeler, Jason W.↗