Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 559 records · Page 31

System and Method for Monitoring Distributed Asset Data

A computer-based monitoring system and monitoring method implemented in computer software for detecting, estimating, and reporting the condition states, their changes, and anomalies for many assets. The assets are of same type, are operated over a period of time, and outfitted with data collection systems. The proposed monitoring method accounts for variability of working conditions for each asset by using regression model that characterizes asset performance. The assets are of the same type but not identical. The proposed monitoring method accounts for asset-to-asset variability; it also accounts for drifts and trends in the asset condition and data. The proposed monitoring system can perform distributed processing of massive amounts of historical data without discarding any useful information where moving all the asset data into one central computing system might be infeasible. The overall processing is includes distributed preprocessing data records from each asset to produce compressed data.

Gorinevsky, Dimitry↗

Deployable sensor system using mesh networking and satellite communication

A sensor system may be configured for continuous operation in a low resource environment and/or in extreme environmental conditions. The sensor system may have sufficient processing capabilities to provide scientific computing for pre-processing, quality control, statistical analysis, event classification, data compression and corrections (e.g., spikes in the data), autonomous decisions and actions, triggering other nodes, and information assurance functions that provide data confidentiality, data integrity, authentication, and non-repudiation. The hardware may have both mesh networking and satellite and cellular communication capability, and may be available for relatively low cost. Such a network provides the flexibility to have potentially any number of nodes be completely independent from one another. Thus, the network may scale across a diverse terrain.

97 MATHEMATICS AND COMPUTING↗

Deployable sensor system using mesh networking and satellite communication

A sensor system may be configured for continuous operation in a low resource environment and/or in extreme environmental conditions. The sensor system may have sufficient processing capabilities to provide scientific computing for pre-processing, quality control, statistical analysis, event classification, data compression and corrections (e.g., spikes in the data), autonomous decisions and actions, triggering other nodes, and information assurance functions that provide data confidentiality, data integrity, authentication, and non-repudiation. The hardware may have both mesh networking and satellite and cellular communication capability, and may be available for relatively low cost. Such a network provides the flexibility to have potentially any number of nodes be completely independent from one another. Thus, the network may scale across a diverse terrain.

Frigo, Janette↗

S-Preconditioner for Multi-fold Data Reduction with Guaranteed User-Controlled Accuracy

The growing gap between the massive amounts of data generated by petascale scientific simulation codes and the capability of system hardware and software to effectively analyze this data necessitates data reduction. Yet, the increasing data complexity challenges most, if not all, of the existing data compression methods. In fact, lossless compression techniques offer no more than 10% reduction on scientific data that we have experience with, which is widely regarded as effectively incompressible. To bridge this gap, in this paper, we advocate a transformative strategy that enables fast, accurate, and multi-fold reduction of double-precision floating-point scientific data. The intuition behind our method is inspired by an effective use of preconditioners for linear algebra solvers optimized for a particular class of computational dwarfs (e.g., dense or sparse matrices). Focusing on a commonly used multi-resolution wavelet compression technique as the underlying solver for data reduction we propose the S-preconditioner, which transforms scientific data into a form with high global regularity to ensure a significant decrease in the number of wavelet coefficients stored for a segment of data. Combined with the subsequent EQ-calibrator, our resultant method (called S-Preconditioned EQ-Calibrated Wavelets (SPEQC-WAVELETS)), robustly achieved a 4- to 5- fold data reduction while guaranteeing user-defined accuracy of reconstructed data to be within 1% point-by-point relative error, lower than 0:01 Normalized RMSE, and higher than 0:99 Pearson Correlation. In this paper, we show the results we obtained by testing our method on six petascale simulation codes including fusion, combustion, climate, astrophysics, and subsurface groundwater in addition to 13 publicly available scientific datasets. We also demonstrate that application-driven data mining tasks performed on decompressed variables or their derived quantities produce results of comparable quality with the ones for the original data.

Jin, Ye↗

Lossless compression of AVIRIS data: Comparison of methods and instrument constraints

A family of lossless compression methods, allowing exact image reconstruction, are evaluated for compressing Airborne Visible/Infrared Imaging Spectrometers (AVIRIS) image data. The methods are used on Differential Pulse Code Modulation (DPCM). The compressed data have an entropy of order 6 bits/pixel. A theoretical model indicates that significantly better lossless compression is unlikely to be achieved because of limits caused by the noise in the AVIRIS channels. AVIRIS data differ from data produced by other visible/near-infrared sensors, such as LANDSAT-TM or SPOT, in several ways. Firstly, the data are recorded at a greater resolution (12 bits, though packed into 16-bit words). Secondly, the spectral channels are relatively narrow and provide continuous coverage of the spectrum so that the data in adjacent channels are generally highly correlated. Thirdly, the noise characteristics of the AVIRIS are defined by the channels' Noise Equivalent Radiances (NER's), and these NER's show that, at some wavelengths, the least significant 5 or 6 bits of data are essentially noise.

Roger, R. E.↗

SnowPac : a multiscale cubic B-spline wavelet compressor for astronomical images

ABSTRACT As more advanced and complex survey telescopes are developed, the size and scale of data being captured grows at increasing rates. Across various domains, data compression through wavelets has enabled the reduction of data size and increase in computation efficiency. In this paper, we provide qualitative and quantitative tests of a new wavelet-based image compression method compared against the current standard for astronomical images. The analysis is improved by making use of state-of-the-art object detection systems to accurately measure the impact of the compression. We find that a combination of lossy wavelet-based methods, efficient quantization, and lossless dictionary compressors can preserve up to 98 per cent of astronomical objects at a 10:1 compression ratio. This significant reduction in file size also preserves astronomical object properties better than existing methods. These methods help further reduce future workloads for image-heavy processing pipelines.

Pulido, Jesus↗

Impacts of swinging door lossy compression of synchrophasor data

As the number of operational Phasor Measurement Units (PMUs) is increasing, compression is becoming crucial for efficiently transferring and storing enormous amount of synchrophasor data continuously measured from a power system. Some utilities in the U.S. have already adopted a nonlinear compression technique, called Swinging Door Algorithm (SDA), assuming compressed data meets technical and regulatory requirements without resulting in any degradation of the measurements. This paper studies impacts of this compression algorithm on field measured and simulated PMU data, and presents results from spectral analyses of raw and decompressed measurements. The results show blind application of the SDA compression alters frequency content in the measurements and introduces false harmonics and aliasing. These can lead to false conclusions being drawn from the measurements about the power system condition.

42 ENGINEERING↗

A bandwidth-compressive modulation system

The phrase 'bandwidth-compressive modulation' means that the compression is achieved not by removing redundancy from the data (data compression), but rather by modulating the carrier phase and amplitude with blocks of data bits. The modulation described here is quadrature amplitude shift keying (QASK), a logical extension of quadrature phase shift keying, (QPSK, or quadriphase). QASK is a cost-effective form of combined phase-and-amplitude modulation. That is, n-bit QASK, like most phase-amplitude forms of modulation, requires considerably less transmitter power than comparably performing n-PSK, while among the class of all n-bit phase-modulation types, n-bit QASK performs nearly optimally but requires the least circuit complexity. A simple, unique microwave transmitter is described which provides high-rate QASK modulation with 4:1 bandwidth compression. A suppressed carrier multiple-loop QASK receiver provides the necessary phase tracking, automatic gain control (AGC), and symbol timing.

Smith, J. G.↗

A multiresolution adaptive wavelet method for nonlinear partial differential equations

We report the multiscale complexity of modern problems in computational science and engineering can prohibit the use of traditional numerical methods in multi-dimensional simulations. Therefore, novel algorithms are required in these situations to solve partial differential equations (PDEs) with features evolving on a wide range of spatial and temporal scales. To meet these challenges, we present a multiresolution wavelet algorithm to solve PDEs with significant data compression and explicit error control. We discretize in space by projecting fields and spatial derivative operators onto wavelet basis functions. We provide error estimates for the wavelet representation of fields and their derivatives. Then, our estimates are used to construct a sparse multiresolution discretization which guarantees the prescribed accuracy. Additionally, we embed a predictor-corrector procedure within the temporal integration to dynamically adapt the computational grid and maintain the accuracy of the solution of the PDE as it evolves. We present examples to highlight the accuracy and adaptivity of our approach.

97 MATHEMATICS AND COMPUTING↗

Bayesian Linear Regression for Hugoniot Data

This repository provides the code and datasets used in the paper Bayesian Analysis of Linear Shock Compression Data. This paper analyzes publicly available shock compression datasets on copper, argon, and nickel from Marsh (1980) using Bayesian linear regression, and compares the results with those obtained using bootstrapping methods. References: - Marsh, S. P. (1980). LASL shock Hugoniot data (Vol. 5). Univ of California Press.

Bernstein, JasonA [Lawrence Livermore National Lab↗

SNAPRed: Reduction of multidimensional neutron time-of-flight diffraction data

SNAP is a neutron time-of-flight diffractometer at the Spallation Neutron Source operated by Oak Ridge National Laboratory. It generates large arrays of neutron detection events that encode the crystalline atomic structure of materials under study. SNAPRed is an application that makes these datasets accessible to end users by orchestrating the process of data reduction while automatically managing the variable neutron instrumentation configuration. It supports arbitrary grouping and masking of individual detector pixels and includes custom-developed data compression approaches to accommodate the large volumes of data generated by the SNAP instrument.

Diffraction↗

Multilevel Techniques for Compression and Reduction of Scientific Data---The Multivariate Case

We develop a technique for multigrid adaptive reduction of data (MGARD). Special attention is given to the case of tensor product grids, where our approach permits the use of nonuniformly spaced grids in each direction, which can prove problematic for many types of data reduction methods. An important feature of our approach is the provision of guaranteed, computable bounds on the loss incurred by the reduction of the data. Many users are leery of lossy algorithms and will only consider using them provided that numerical bounds on the pointwise difference between the original and the reduced datasets are given. Accordingly, we develop techniques for bounding the loss measured in the $L^{\infty}(\Omega)$ norm, and we show that these bounds are realistic in the sense that they do not significantly overestimate the actual loss. Here, the resulting loss indicators are used to guide the adaptive reduction of the data so that the reduced dataset meets a user-prescribed tolerance or memory constraint. Illustrative numerical examples, including the reduction of data arising from the simulation of a nonlinear reaction-diffusion problem, a turbulent channel flow, and a climate simulation, are provided.

97 MATHEMATICS AND COMPUTING↗

HBMax: Optimizing Memory Efficiency for Parallel Influence Maximization on Multicore Architectures

The goal of influence maximization is to select k most-influential vertices or seeds in a network, where influence is defined by a given diffusion process. The problem has a number of important applications such as viral marketing, information spread, and epidemic control. Although computing optimal seed set is NP-Hard, due to the submodular nature of the problem efficient approximation algorithms exist. However, even state-of-the-art parallel implementations are limited by a sampling step that incurs large memory footprints. This in turn limits the problem size reach and approximation quality. In this work, we study the memory footprint of the sampling process collecting reverse reachability information in the IMM algorithm over large real-world social networks. We present an adaptive and memory-efficient optimization approach for a state-of-the-art multi-threaded parallel influence maximization algorithm. Our approach,HuffMax, uses a portion of the reverse reachable (RR) sets collected by the algorithm to learn the characteristics of the graph. Then, it compresses the intermediate reverse reachability information with Huffman coding, and queries directly on the compressed data to preserve the memory savings obtained through compression. We also propose an efficient sampling strategy based on the distribution of RR sets, which can further reduce the computation time for typical social networks with long-tail distributions. Considering a NUMA architecture, we scale up our solution on 128-core CPUs and reduce the memory footprint by up to 45.7% with negligible time overhead (or even faster) and without perceivable loss of accuracy.

Chen, Xinyu↗

buhito

buhito is a Python library for graph analysis and machine learning. Graphs can represent networks with objects as nodes and their relationships as edges. buhito focuses on graphlet methods that study graphs through enumerating their component subgraphs to enable interpretable and fast models of complex systems. The package provides tools for different algorithmic designs for computing, analyzing, and applying graphlets to research problems such as machine learning, data compression, and anomaly detection in graph-structured data. A central feature is performing decomposition data analysis on graphs for machine learning models. Implemented in Python and built upon open-source scientific libraries such as NetworkX, NumPy, and SciPy, buhito provides high-performance methods for researchers exploring the mathematical and computational foundations of graphlet analysis applicable to systems of different sizes.

Pimonova, Yulia↗

Spatial degradation of satellite data

Consideration is given to a technique for spatially degrading high-resolution satellite data to produce comparable data sets over a range of coarser resolutions. Landsat MSS data is used to produce seven spatial resolution data sets by applying a spatial filter designed to simulate sensor response. Also, spatial degradation of coarse resolution data to provide data compression for the production of global-scale data sets is examined. NOAA AVHRR Global Area Coverage data is compared to other sampling procedures. It is found that sampling procedures that incorporate averaging result in decreased variance, while sampling procedures adopting single-value selection have higher variances and produce data values comparable with those from the original data.

Justice, C. O.↗

Method of Real-Time Principal-Component Analysis

Dominant-element-based gradient descent and dynamic initial learning rate (DOGEDYN) is a method of sequential principal-component analysis (PCA) that is well suited for such applications as data compression and extraction of features from sets of data. In comparison with a prior method of gradient-descent-based sequential PCA, this method offers a greater rate of learning convergence. Like the prior method, DOGEDYN can be implemented in software. However, the main advantage of DOGEDYN over the prior method lies in the facts that it requires less computation and can be implemented in simpler hardware. It should be possible to implement DOGEDYN in compact, low-power, very-large-scale integrated (VLSI) circuitry that could process data in real time.

Duong, Tuan↗

Real-Time Principal-Component Analysis

A recently written computer program implements dominant-element-based gradient descent and dynamic initial learning rate (DOGEDYN), which was described in Method of Real-Time Principal-Component Analysis (NPO-40034) NASA Tech Briefs, Vol. 29, No. 1 (January 2005), page 59. To recapitulate: DOGEDYN is a method of sequential principal-component analysis (PCA) suitable for such applications as data compression and extraction of features from sets of data. In DOGEDYN, input data are represented as a sequence of vectors acquired at sampling times. The learning algorithm in DOGEDYN involves sequential extraction of principal vectors by means of a gradient descent in which only the dominant element is used at each iteration. Each iteration includes updating of elements of a weight matrix by amounts proportional to a dynamic initial learning rate chosen to increase the rate of convergence by compensating for the energy lost through the previous extraction of principal components. In comparison with a prior method of gradient-descent-based sequential PCA, DOGEDYN involves less computation and offers a greater rate of learning convergence. The sequential DOGEDYN computations require less memory than would parallel computations for the same purpose. The DOGEDYN software can be executed on a personal computer.

Duong, Vu↗

Hybrid learning techniques for scientific data reduction with performance guarantees

The research initiatives supported by the U.S. Department of Energy (DOE) Grant DE-SC0022265 are fundamentally aimed at pioneering advanced machine learning (ML) techniques for scientific data compression within high-performance computing (HPC) environments. This comprehensive body of work addresses the critical challenge posed by the exponential growth of data generated by scientific simulations in domains such as fusion energy, climate modeling, and computational fluid dynamics (CFD). A core objective is to develop compression algorithms that achieve substantial data reduction—often by orders of magnitude—while rigorously ensuring the fidelity of both the primary data (PD) and scientifically crucial derived quantities of interest (QoI). The methodologies deployed under this grant integrate sophisticated deep learning architectures, prominently featuring autoencoders, advanced generative models like conditional diffusion, and hybrid learning techniques. Key innovations include the development of Guaranteed Autoencoders (GAE) and the Guaranteed Conditional Diffusion with Tensor Correction (GCDTC) framework, which provide explicit, instance-level error bounds on reconstructed data. Furthermore, specialized strategies such as nonlinear constraint satisfaction are employed to preserve the integrity of QoI, a vital requirement for the trustworthiness of downstream scientific analyses. This research also focuses on the design and implementation of scalable, GPU-accelerated software pipelines that seamlessly integrate into existing HPC workflows, ensuring both computational efficiency and practical applicability. The CAESAR framework, for example, unifies foundation and generative models to create an adaptive and efficient compression solution for spatio-temporal scientific data. Collectively, these efforts represent a significant advancement in mitigating the scientific data deluge, enabling more effective data management, accelerated scientific discovery, and optimized utilization of HPC resources.

97 MATHEMATICS AND COMPUTING↗