Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 811 records · Page 45
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Massive compression for high data rate macromolecular crystallography (HDRMX): impact on diffraction data and subsequent structural analysis

New higher-count-rate, integrating, large-area X-ray detectors with framing rates as high as 17400 images per second are beginning to be available. These will soon be used for specialized macromolecular crystallography experiments but will require optimal lossy compression algorithms to enable systems to keep up with data throughput. Some information may be lost. Can we minimize this loss with acceptable impact on structural information? To explore this question, we have considered several approaches: summing short sequences of images, binning to create the effect of larger pixels, use of JPEG-2000 lossy wavelet-based compression, and use of Hcompress, which is a Haar-wavelet-based lossy compression borrowed from astronomy. We also explore the effect of the combination of summing, binning, and Hcompress or JPEG-2000. In each of these last two methods one can specify approximately how much one wants the result to be compressed from the starting file size. These provide particularly effective lossy compressions that retain essential information for structure solution from Bragg reflections.

47 OTHER INSTRUMENTATION↗
DOE OSTI2024

Automated pipeline processing X-ray diffraction data from dynamic compression experiments on the Extreme Conditions Beamline of PETRA III

Presented and discussed here is the implementation of a software solution that provides prompt X-ray diffraction data analysis during fast dynamic compression experiments conducted within the dynamic diamond anvil cell technique. It includes efficient data collection, streaming of data and metadata to a high-performance cluster (HPC), fast azimuthal data integration on the cluster, and tools for controlling the data processing steps and visualizing the data using the DIOPTAS software package. This data processing pipeline is invaluable for a great number of studies. The potential of the pipeline is illustrated with two examples of data collected on ammonia–water mixtures and multiphase mineral assemblies under high pressure. The pipeline is designed to be generic in nature and could be readily adapted to provide rapid feedback for many other X-ray diffraction techniques, e.g. large-volume press studies, in situ stress/strain studies, phase transformation studies, chemical reactions studied with high-resolution diffraction etc.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Optimizing Error-Bounded Lossy Compression for Scientific Data With Diverse Constraints

Vast volumes of data are produced by today's scientific simulations and advanced instruments. These data cannot be stored and transferred efficiently because of limited I/O bandwidth, network speed, and storage capacity. Error-bounded lossy compression can be an effective method for addressing these issues: not only can it significantly reduce data size, but it can also control the data distortion based on user-defined error bounds. In practice, many scientific applications have specific requirements or constraints for lossy compression, in order to guarantee that the reconstructed data are valid for post hoc analysis. For example, some datasets contain irrelevant data that should be isolated in particular and users often have intuition regarding value ranges, geospatial regions, and other data subsets that are crucial for subsequent analysis. Existing state-of-the-art error-bounded lossy compressors, however, do not consider these constraints during compression, resulting in inferior compression ratios with respect to user's post hoc analysis, due to the fact that the data itself provides little or no value for post hoc analysis. In this work we address this issue by proposing an optimized framework that can preserve diverse constraints during the error-bounded lossy compression, e.g., cleaning the irrelevant data, efficiently preserving different precision for multiple value intervals, and allowing users to set diverse precision over both regular and irregular regions. We perform our evaluation on a supercomputer with up to 2,100 cores. Experiments with six real-world applications show that our proposed diverse constraints based error-bounded lossy compressor can obtain a higher visual quality or data fidelity on reconstructed data with the same or even higher compression ratios compared with the traditional state-of-the-art compressor SZ. Furthermore, our experiments also demonstrate very good scalability in compression performance compared with the I/O throughput of the parallel file system.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1976

Turbulence measurements in a transonic boundary layer and free-shear flow using laser velocimetry and hot-wire anemometry techniques

Quantitative measurements of the turbulence fluctuations in velocity and mass flux have been obtained in Mach 0.6 and 0.8 turbulent boundary layer and free-shear layer flows by laser velocimetry and hot-wire anemometry techniques. To evaluate the effects of compressibility, these transonic data are compared to available incompressible and supersonic results. Based on some simplifying assumptions, estimates of the rms density fluctuations are made for which error bounds are given. In addition to these fluctuation data, the compressible mean velocity data obtained with the laser velocimeter are presented and compared to pitot tube results. The investigation was conducted in the Ames 6- by 6-Foot Supersonic Wind Tunnel at free-stream Mach numbers of 0.6 and 0.8 for a unit Reynolds number of about 10,000,000 per meter.

Johnson, D. A.↗
DOE OSTI2024

Zero-Power Analog Optical Processing

The motivation behind this research is the growing challenge of handling the massive amounts of data generated by modern imaging systems. Conventional digital image processing techniques are struggling to keep pace with the demands of high-resolution and high-speed imaging systems for remote sensing due to their high-power consumption and data storage requirements. We present a novel approach based on analog photonics to address this challenge. The proposed system utilizes a silicon-photonics-based image encoder positioned after image formation and initial optical-to-electrical conversion. The photonic encoder compresses image data using a passive disordered photonic structure to perform kernel-type random projections of the raw data. The compressed data is then processed by a back-end neural network, which reconstructs the original image with high fidelity (structural similarity exceeding 90%). Our proposed approach has the potential to compress images with ~ 1000X lower power consumption compared to digital approaches with data rates exceeding 1 terapixel/second.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2006

Pre-Processor for Compression of Multispectral Image Data

A computer program that preprocesses multispectral image data has been developed to provide the Mars Exploration Rover (MER) mission with a means of exploiting the additional correlation present in such data without appreciably increasing the complexity of compressing the data.

Klimesh, Matthew↗
DOE OSTI2021

Method and apparatus for using compression to improve performance of low voltage caches

A method of operating a cache in a computing device includes, in response to receiving a memory access request at the cache, determining compressibility of data specified by the request, selecting in the cache a destination portion for storing the data based on the compressibility of the data and a persistent fault history of the destination portion, and storing a compressed copy of the data in a non-faulted subportion of the destination portion, wherein the persistent fault history indicates that the non-faulted subportion excludes any persistent faults.

Kalamatianos, John↗
NASA NTRS2000

Collected Data of The Boreal Ecosystem and Atmosphere Study (BOREAS)

The Boreal Ecosystem-Atmosphere Study (BOREAS) was a large-scale international interdisciplinary climate-ecosystem interaction experiment in the northern boreal forests of Canada. Its goal was to improve our understanding of the boreal forests -- how they interact with the atmosphere, how much CO2 they can store, and how climate change will affect them. BOREAS wanted to learn to use satellite data to monitor the forests, and to improve computer simulation and weather models so scientists can anticipate the effects of global change. This BOREAS CD-ROM set is a set of 12 CD-ROMs containing the finalized point data sets and compressed image data from the BOREAS Project. All point data are stored in ASCII text files, and all image and GIS products are stored as binary images, compressed using GZip. Additional descriptions of the various data sets on this CD-ROM are available in other documents in the BOREAS series.

Newcomer, J.↗
NASA NTRS2001

Compression of echocardiographic scan line data using wavelet packet transform

An efficient compression strategy is indispensable for digital echocardiography. Previous work has suggested improved results utilizing wavelet transforms in the compression of 2D echocardiographic images. Set partitioning in hierarchical trees (SPIHT) was modified to compress echocardiographic scanline data based on the wavelet packet transform. A compression ratio of at least 94:1 resulted in preserved image quality.

Non-NASA Center↗
DOE OSTI2022

Toward Quantity-of-Interest Preserving Lossy Compression for Scientific Data

Today's scientific simulations and instruments are producing a large amount of data, leading to difficulties in storing, transmitting, and analyzing these data. While error-controlled lossy compressors are effective in significantly reducing data volumes and efficiently developing databases for multiple scientific applications, they mainly support error controls on raw data, which leaves a significant gap between the data and user's downstream analysis. This may cause unqualified uncertainties in the outcomes of the analysis, a.k.a quantities of interest (QoIs), which are the major concerns of users in adopting lossy compression in practice. In this paper, we propose rigorous mathematical theories to preserve four families of QoIs that are widely used in scientific analysis during lossy compression along with practical implementations. Specifically, we first develop the error control theory for univariate QoIs which are essential for computing physical properties such as kinetic energy, followed by multivariate QoIs that are more commonly used in real-world applications. The proposed method is integrated into a state-of-the-art compression framework in a modular fashion, which could easily adapt to new QoIs and new compression algorithms. Experiments on real-world datasets demonstrate that the proposed method provides faithful error control on important QoIs including kinetic energy, regional average, and isosurface without trials and errors, while offering compression ratios that are up to 4x of the compression ratios provided by state-of-the-art compressors.

Jiao, Pu↗
DOE OSTI2020

More chemical detection through less sampling: amplifying chemical signals in hyperspectral data cubes through compressive sensing

Compressive sensing (CS) is a method of sampling which permits some classes of signals to be reconstructed with high accuracy even when they were sampled at sub-Nyquist rates. In this paper we explore a phenomenon in which bandwise CS sampling of a hyperspectral data cube followed by reconstruction can actually result in amplification of chemical signals contained in the cube. Perhaps most surprisingly, chemical signal amplification generally seems to increase as the level of sampling decreases. In some examples, the chemical signal is significantly stronger in a data cube reconstructed from 10% CS sampling than it is in the raw, 100% sampled data cube. We explore this phenomenon in two real-world datasets including the Physical Sciences Inc. Fabry-Perot interferometer sensor multispectral dataset and the Johns Hopkins Applied Physics Lab FTIR-based longwave infrared sensor hyperspectral dataset. Each of these datasets contains the release of a chemical simulant, such as glacial acetic acid, triethyl phospate, and sulfur hexafluoride, and in all cases we use the adaptive coherence estimator (ACE) to detect a target signal in the hyperspectral data cube. We end the paper by suggesting some theoretical justifications for why chemical signals would be amplified in CS sampled and reconstructed hyperspectral data cubes and discuss some practical implications.

Hyperspectral imagery, Compressive Sensing, single↗
DOE OSTI2023

Black-box statistical prediction of lossy compression ratios for scientific data

Lossy compressors are increasingly adopted in scientific research, tackling volumes of data from experiments or parallel numerical simulations and facilitating data storage and movement. In contrast with the notion of entropy in lossless compression, no theoretical or data-based quantification of lossy compressibility exists for scientific data. Users rely on trial and error to assess lossy compression performance. As a strong data-driven effort toward quantifying lossy compressibility of scientific datasets, we provide a statistical framework to predict compression ratios of lossy compressors. Our method is a two-step framework where (i) compressor-agnostic predictors are computed and (ii) statistical prediction models relying on these predictors are trained on observed compression ratios. Proposed predictors exploit spatial correlations and notions of entropy and lossyness via the quantized entropy. We study 8+ compressors on 6 scientific datasets and achieve a median percentage prediction error less than 12%, which is substantially smaller than that of other methods while achieving at least a 8.8× speedup for searching for a specific compression ratio and 7.8× speedup for determining the best compressor out of a collection.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS2018

Digital Map Rendering Method

The invention is a system and method of compressing terrain data to be used to render a three-dimensional map representation of the terrain data. In general, the invention operates by first selecting the boundaries of the terrain data and to be compressed and dividing the terrain data into regular geometric areas. Next, a type of free-edged, planar geometric surface is defined which is used to approximate the terrain data for each regular geometric area. The approximations are checked to determine if they fall within user selected tolerances. If the approximation for a specific regular geometric area is within the user specified tolerances, the data is saved for that specific regular geometric area. If the approximation for a specific regular geometric area falls outside the user specified tolerances, the regular geometric area is divided and a free-edged, planar geometric surface approximation is made for each of the divided areas. This process is recursively repeated until all of the regular geometric areas are approximated by free-edged, planar surfaces. Finally, the compressed terrain data is used to render a three-dimensional map.

Skoog, Mark↗
DOE OSTI2021

Simultaneous compression and opacity data from time-series radiography with a Lagrangian marker

Time-resolved radiography can be used to obtain absolute shock Hugoniot states by simultaneously measuring at least two mechanical parameters of the shock, and this technique is particularly suitable for one-dimensional converging shocks where a single experiment probes a range of pressures as the converging shock strengthens. However, at sufficiently high pressures, the shocked material becomes hot enough that the x-ray opacity falls significantly. Additionally, if the system includes a Lagrangian marker such that the mass within the marker is known, this additional information can be used to constrain the opacity as well as the Hugoniot state. In the limit that the opacity changes only on shock heating, and not significantly on subsequent isentropic compression, the opacity of the shocked material can be determined uniquely. More generally, it is necessary to assume the form of the variation of opacity with isentropic compression or to introduce multiple marker layers. Alternatively, assuming either the equation of state or the opacity, the presence of a marker layer in such experiments enables the non-assumed property to be deduced more accurately than from the radiographic density reconstruction alone. An example analysis is shown for measurements of a converging shock wave in polystyrene at the National Ignition Facility.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
NASA NTRS1992

Navier-Stokes analysis and experimental data comparison of compressible flow within ducts

Many aircraft employ ducts with centerline curvature or changing cross-sectional shape to join the engine with inlet and exhaust components. S-ducts convey air to the engine compressor from the intake and often decelerate the flow to achieve an acceptable Mach number at the engine compressor by increasing the cross-sectional area downstream. Circular-to-rectangular transition ducts are used on aircraft with rectangular exhaust nozzles to connect the engine and nozzle. To achieve maximum engine performance, the ducts should minimize flow total pressure loss and total pressure distortion at the duct exit. Changes in the curvature of the duct centerline or the duct cross-sectional shape give rise to streamline curvature which causes cross stream pressure gradients. Secondary flows can be caused by deflection of the transverse vorticity component of the boundary layer. This vortex tilting results in counter-rotating vortices. Additionally, the adverse streamwise pressure gradient caused by increasing cross-sectional area can lead to flow separation. Vortex pairs have been observed in the exit planes of both duct types. These vortices are due to secondary flows induced by pressure gradients resulting from streamline curvature. Regions of low total pressure are produced when the vortices convect boundary layer fluid into the main flow. The purpose of the present study is to predict the measured flow field in a diffusing S-duct and a circular-to-rectangular transition duct with a full Navier-Stokes computer program, PARC3D, and to compare the numerical predictions with new detailed experimental measurements. The work was undertaken to extend previous studies and to provide additional CFD validation data needed to help model flows with strong secondary flow and boundary layer separation. The S-duct computation extends the study of Smith et al, and Harloff et al, which concluded that the computation might be improved by using a finer grid and more advanced turbulence models. The present study compares results for both the Baldwin-Lomas and k-epsilon turbulence models and is conducted with a refined grid. For the transition duct, two inlet conditions were considered, the first with straight flow and the second with swirling flow. The first case permits examination of the effects of the geometric transition on the flow field, while the second case includes the rotational flow effect characteristic of a gas turbine engine.

Harloff, G. J.↗