Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,045 records · Page 58
All sourcesNASA NTRSDOE OSTI
NASA NTRS2000

BOREAS RSS-16 AIRSAR CM Images: Integrated Processor Version 6.1 Level-3b

The BOREAS RSS-16 team used satellite and aircraft SAR data in conjunction with various ground measurements to determine the moisture regime of the boreal forest. RSS-16 assisted with the acquisition and ordering of NASA JPL AIRSAR data collected from the NASA DC-8 aircraft. The NASA JPL AIRSAR is a side-looking imaging radar system that utilizes the SAR principle to obtain high resolution images that represent the radar backscatter of the imaged surface at different frequencies and polarizations. The information contained in each pixel of the AIRSAR data represents the radar backscatter for all possible combinations of horizontal and vertical transmit and receive polarizations (i.e., HH, HV, VH, and VV). Geographically, the data cover portions of the BOREAS SSA and NSA. Temporally, the data were acquired from 12-Aug-1993 to 31-Jul-1995. The level-3b AIRSAR CM data are in compressed Stokes matrix format, which has 10 bytes per pixel. From this data format, it is possible to synthesize a number of different radar backscatter measurements. The data are stored in binary image-format files. The data files are available on a CD-ROM (see document number 20010000884), or from the Oak Ridge National Laboratory (ORNL) Distributed Active Archive Center (DAAC).

Hall, Forrest G.↗
NASA NTRS2003

Reliable Transport over SpaceWire for James Webb Space Telescope (JWST) Focal Plane Electronics (FPE) Network

NASA's James Webb Space Telescope (JWST) faces difficult technical and budgetary challenges to overcome before it is scheduled launch in 2010. The Integrated Science Instrument Module (ISIM), shares these challenges. The major challenge addressed in this paper is the data network used to collect, process, compresses and store Infrared data. A total of 114 Mbps of raw information must be collected from 19 sources and delivered to the two redundant data processing units across a twenty meter deployed thermally restricted interface. Further data must be transferred to the solid-state recorder and the spacecraft. The JWST detectors are kept at cryogenic temperatures to obtain the sensitivity necessary to measure faint energy sources. The Focal Plane Electronics (FPE) that sample the detector, generate packets from the samples, and transmit these packets to the processing electronics must dissipate little power in order to help keep the detectors at these cold temperatures. Separating the low powered front-end electronics from the higher-powered processing electronics, and using a simple high-speed protocol to transmit the detector data minimize the power dissipation near the detectors. Low Voltage Differential Signaling (LVDS) drivers were considered an obvious choice for physical layer because of their high speed and low power. The mechanical restriction on the number cables across the thermal interface force the Image packets to be concentrated upon two high-speed links. These links connect the many image packet sources, Focal Plane Electronics (FPE), located near the cryogenic detectors to the processing electronics on the spacecraft structure. From 12 to 10,000 seconds of raw data are processed to make up an image, various algorithms integrate the pixel data Loss of commands to configure the detectors as well as the loss of science data itself may cause inefficiency in the use of the telescope that are unacceptable given the high cost of the observatory. This combination of requirements necessitates a redundant, fault tolerant, high- speed, low mass, low power network with a low Bit error Rate(1E-9- 1E-12). The ISIM systems team performed many studies of the various network architectures that meeting these requirements. The architecture selected uses the Spacewire protocol, with the addition of a new transport and network layer added to implement end-to-end reliable transport. The network and reliable transport mechanism must be implemented in hardware because of the high average information rate and the restriction on the ability of the detectors to buffer data due to power and size restrictions. This network and transport mechanism was designed to be compatible with existing Spacewire links and routers so that existing equipment and designs may be leveraged upon. The transport layer specification is being coordinated with European Space Agency (ESA), Spacewire Working Group and the Consultative Committee for Space Data System (CCSDS) PlK Standard Onboard Interface (SOIF) panel, with the intent of developing a standard for reliable transport for Spacewire. Changes to the protocol presented are likely since negotiations are ongoing with these groups. A block of RTL VHDL that implements a multi-port Spacewire router with an external user interface will be developed and integrated with an existing Spacewire Link design. The external user interface will be the local interface that sources and sinks packets onto and off of the network (Figure 3). The external user interface implements the network and transport layer and handles acknowledgements and re-tries of packets for reliable transport over the network. Because the design is written in RTL, it may be ported to any technology but will initially be targeted to the new Actel Accelerator series (AX) part. Each link will run at 160 Mbps and the power will be about 0.165 Watt per link worst case in the Actel AX.

Rakow, Glenn↗
NASA NTRS2013

Usage of Data-Encoded Web Maps with Client Side Color Rendering for Combined Data Access, Visualization and Modeling Purposes

Current approaches to satellite observation data storage and distribution implement separate visualization and data access methodologies which often leads to the need in time consuming data ordering and coding for applications requiring both visual representation as well as data handling and modeling capabilities. We describe an approach we implemented for a data-encoded web map service based on storing numerical data within server map tiles and subsequent client side data manipulation and map color rendering. The approach relies on storing data using the lossless compression Portable Network Graphics (PNG) image data format which is natively supported by web-browsers allowing on-the-fly browser rendering and modification of the map tiles. The method is easy to implement using existing software libraries and has the advantage of easy client side map color modifications, as well as spatial subsetting with physical parameter range filtering. This method is demonstrated for the ASTER-GDEM elevation model and selected MODIS data products and represents an alternative to the currently used storage and data access methods. One additional benefit includes providing multiple levels of averaging due to the need in generating map tiles at varying resolutions for various map magnification levels. We suggest that such merged data and mapping approach may be a viable alternative to existing static storage and data access methods for a wide array of combined simulation, data access and visualization purposes.

Pliutau, Denis↗
NASA NTRS2008

Compression After Impact Testing of Sandwich Structures Using the Four Point Bend Test

For many composite laminated structures, the design is driven by data obtained from Compression after Impact (CAI) testing. There currently is no standard for CAI testing of sandwich structures although there is one for solid laminates of a certain thickness and lay-up configuration. Most sandwich CAI testing has followed the basic technique of this standard where the loaded ends are precision machined and placed between two platens and compressed until failure. If little or no damage is present during the compression tests, the loaded ends may need to be potted to prevent end brooming. By putting a sandwich beam in a four point bend configuration, the region between the inner supports is put under a compressive load and a sandwich laminate with damage can be tested in this manner without the need for precision machining. Also, specimens with no damage can be taken to failure so direct comparisons between damaged and undamaged strength can be made. Data is presented that demonstrates the four point bend CAI test and is compared with end loaded compression tests of the same sandwich structure.

Nettles, Alan T.↗
DOE OSTI2025

What to Support When You’re Compressing

Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications’ data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications’ needs and compressors’ capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.

Error-Bounded Lossy Compression↗
DOE OSTI2021

Equation-of-state, sound speed, and reshock of shock-compressed fluid carbon dioxide

Mechanical equation-of-state data of initially liquid and solid CO2 shock-compressed to terapascal conditions are reported. Diamond-sapphire anvil cells were used to vary the initial density and state of CO2 samples that were then further compressed with laser-driven shock waves, resulting in a data set from which precise derivative quantities, including Grüneisen parameter and sound speed, are determined. Reshock states are measured to 800 GPa and map the same pressure-density conditions as the single shock using different thermodynamic paths. The compressibility data reported here do not support current density-functional-theory calculations, but are better represented by tabular equation-of-state models.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

StOKeDMD: Streaming Occupation kernel dynamic mode decomposition

Dynamic mode decomposition (DMD) has become a common technique for constructing surrogate models for dynamical systems from observed system states. The Occupation Kernel DMD (OKDMD) method proposed in (Rosenfeld et al., 2022) and (Rosenfeld et al., 2024) is a Liouville operator based method that builds surrogate models from system state trajectories. Here, this paper proposes an extension of OKDMD to the case when the system states are observed in a streaming fashion, i.e., only a small fraction of the state trajectory is available at a given time. The developed method, Streaming Occupation Kernel DMD (StOKeDMD), accommodates the streaming data input by leveraging properties of specific choices of kernel functions and occupation kernels. We apply the StoKeDMD method as a compression method for streaming data, analyze the memory complexity, and demonstrate the performance of StoKeDMD in the compression of streaming data generated from a Lorenz system and a fluid flow simulation.

97 MATHEMATICS AND COMPUTING↗
NASA NTRS1986

Micromechanics of composite laminate compression failure

The Dugdale analysis for metals loaded in tension was adapted to model the failure of notched composite laminates loaded in compression. Compression testing details, MTS alignment verification, and equipment needs were resolved. Thus far, only 2 ductile material systems, HST7 and F155, were selected for study. A Wild M8 Zoom Stereomicroscope and necessary attachments for video taping and 35 mm pictures were purchased. Currently, this compression test system is fully operational. A specimen is loaded in compression, and load vs shear-crippling zone size is monitored and recorded. Data from initial compression tests indicate that the Dugdale model does not accurately predict the load vs damage zone size relationship of notched composite specimens loaded in compression.

Guynn, E. Gail↗
NASA NTRS2012

Development of RGB Composite Imagery for Operational Weather Forecasting Applications

The NASA Short-term Prediction Research and Transition (SPoRT) Center, in collaboration with the Cooperative Institute for Research in the Atmosphere (CIRA), is providing red-green-blue (RGB) color composite imagery to several of NOAA s National Centers and National Weather Service forecast offices as a demonstration of future capabilities of the Advanced Baseline Imager (ABI) to be implemented aboard GOES-R. Forecasters rely upon geostationary satellite imagery to monitor conditions over their regions of responsibility. Since the ABI will provide nearly three times as many channels as the current GOES imager, the volume of data available for analysis will increase. RGB composite imagery can aid in the compression of large data volumes by combining information from multiple channels or paired channel differences into single products that communicate more information than provided by a single channel image. A standard suite of RGB imagery has been developed by the European Organization for the Exploitation of Meteorological Satellites (EUMETSAT), based upon the Spinning Enhanced Visible and Infrared Imager (SEVIRI). The SEVIRI instrument currently provides visible and infrared wavelengths comparable to the future GOES-R ABI. In addition, the Moderate Resolution Imaging Spectroradiometer (MODIS) instruments aboard the NASA Terra and Aqua satellites can be used to demonstrate future capabilities of GOES-R. This presentation will demonstrate an overview of the products currently disseminated to SPoRT partners within the GOES-R Proving Ground, and other National Weather Service forecast offices, along with examples of their application. For example, CIRA has used the channels of the current GOES sounder to produce an "air mass" RGB originally designed for SEVIRI. This provides hourly imagery over CONUS for looping applications while demonstrating capabilities similar to the future ABI instrument. SPoRT has developed similar "air mass" RGB imagery from MODIS, and through a case study example, synoptic-scale features evident in single-channel water vapor imagery are shown in the context of the air mass product. Other products, such as the "nighttime microphysics" RGB, are useful in the detection of low clouds and fog. Nighttime microphysics products from MODIS offer some advantages over single-channel or spectral difference techniques and will be discussed in the context of a case study. Finally, other RGB products from SEVIRI are being demonstrated as precursors to GOES-R within the GOES-R Proving Ground. Examples of "natural color" and "dust" imagery will be shown with relevant applications.

Molthan, Andrew L.↗
DOE OSTI2025

Optimising the processing and storage of visibilities using lossy compression

The next-generation radio astronomy instruments are providing a massive increase in sensitivity and coverage, largely through increasing the number of stations in the array and the frequency span sampled. The two primary problems encountered when processing the resultant avalanche of data are the need for abundant storage and the constraints imposed by I/O, as I/O bandwidths drop significantly on cold storage. An example of this is the data deluge expected from the SKA Telescopes of more than 60 PB per day, all to be stored on the buffer filesystem. While compressing the data is an obvious solution, the impacts on the final data products are hard to predict. In this paper, we chose an error-controlled compressor – MGARD – and applied it to simulated SKA-Mid and real pathfinder visibility data, in noise-free and noise-dominated regimes. As the data have an implicit error level in the system temperature, using an error bound in compression provides a natural metric for compression. MGARD ensures the compression incurred errors adhere to the user-prescribed tolerance. To measure the degradation of images reconstructed using the lossy compressed data, we proposed a list of diagnostic measures, exploring the trade-off between these error bounds and the corresponding compression ratios, as well as the impact on science quality derived from the lossy compressed data products through a series of experiments. We studied the global and local impacts on the output images for continuum and spectral line examples. We found relative error bounds of as much as 10%, which provide compression ratios of about 20, have a limited impact on the continuum imaging as the increased noise is less than the image RMS, whereas a 1% error bound (compression ratio of 8) introduces an increase in noise of about an order of magnitude less than the image RMS. For extremely sensitive observations and for very precious data, we would recommend a 0.1% error bound with compression ratios of about 4. These have noise impacts two orders of magnitude less than the image RMS levels. At these levels, the limits are due to instabilities in the deconvolution methods. We compared the results to the alternative compression tool DYSCO, in both the impacts on the images and in the relative flexibility. MGARD provides better compression for similar error bounds and has a host of potentially powerful additional features.

Techniques: interferometric↗
NASA NTRS1994

Progressive Vector Quantization on a massively parallel SIMD machine with application to multispectral image data

A progressive vector quantization (VQ) compression approach is discussed which decomposes image data into a number of levels using full search VQ. The final level is losslessly compressed, enabling lossless reconstruction. The computational difficulties are addressed by implementation on a massively parallel SIMD machine. We demonstrate progressive VQ on multispectral imagery obtained from the Advanced Very High Resolution Radiometer instrument and other Earth observation image data, and investigate the trade-offs in selecting the number of decomposition levels and codebook training method.

Manohar, Mareboyana↗
NASA NTRS1960

Correlation of the Buckling Strength of Pressurized Cylinders in Compression or Bending with Structural Parameters

The data on nonpressurized cylinders in bending of NACA TN 3735 and the data on pressurized cylinders in compression and bending of NASA TN D-360 are correlated with structural parameters by using small-deflection buckling theory and reduced values for the extensional stiffness of the cylinder wall. The correlating procedure should prove useful in future shell-buckling investigations by reducing the number of tests required.

Peterson, James P.↗
DOE OSTI2023

HarDWR - Water Management Area (WMA) Shapefiles

For a detailed description of the database of which this record is only one part, please see the HarDWR meta-record. Borders of all Water Management Areas (WMAs) across the 11 Western-most states of the continental United States are available filtered through a single source. The legal name for this set of boundaries varies state-by-state. The data is provided as two compressed shapefiles. One, stateWMAs, contains data for all 11 states. For 10 of those states, Arizona being the exception, the polygons represent the legal management boundaries used by those states to manage their surface and groundwater resources respectively. Arizona is unique among this collection of states in that surface and groundwater resources are managed using two separate sets of boundaries. During our followup analysis, due to technical reasons we decided to focus on one set of boundaries, those for surface water. Due to this, the Arizona surface WMAs are included within stateWMAs. The Arizona groundwater WMAs are provided as the second file azGroundWMAs as a companion to the first file for completeness and general reference, although the boundaries were no longer referenced in the analysis (Grogan et al., in review) past a certain point. The columns for both shapefiles are: basinNum: the state provided unique numerical ID basinName: the state provided English name of the area, where applicable state: the state name uniID: a unique identifier we created by concatenating the state name, and underscore, and the state numerical ID

Economics↗
DOE OSTI2020

HarDWR - Water Management Area (WMA) Shapefiles

A dataset within the Harmonized Database of Western U.S. Water Rights (HarDWR). For a detailed description of the database, please see the meta-record v2.0. Changelog v2.0 - No changes v1.0 - Initial public release Description Borders of all Water Management Areas (WMAs) across the 11 western-most states of the coterminous United States are available filtered through a single source. The legal name for this set of boundaries varies state-by-state. The data is provided as two compressed shapefiles. One, stateWMAs, contains data for all 11 states. For 10 of those states, Arizona being the exception, the polygons represent the legal management boundaries used by those states to manage their surface and groundwater resources respectively. WMAs refer to the set of boundaries a particular state uses to manage its water resources. Each set of boundaries was collected from the states individually, and then merged into one spatial layer. The merging process included renaming some columns to enable merging with all other source layers, as well as removing columns deemed not required for followup analysis. The retained columns for each boundary are: basinNum - the state provided unique numerical ID; basinName - the state provided English name of the area, where applicable; state - the state name; and uniID - a unique identifier we created by concatenating the state name, and underscore, and the state numerical ID. Arizona is unique within this collection of states in that surface and groundwater resources are managed using two separate sets of boundaries. During our followup analysis (Grogan et al., in review) we decided to focus on one set of boundaries, those for surface water. This is due to the recommendation of our hydrologists that the surface water boundary set is a more realist representation of how water moves across the landscape, as a few of the groundwater boundaries are based on political and/or economic considerations. Therefore, the Arizona surface WMAs are included within stateWMAs. The Arizona groundwater WMAs are provided as a separate file, azGroundWMAs, as a companion to the first file for completeness and general reference. WMA spatial boundary data sources by state: Arizona: Arizona Surface Water Watersheds; Collected February, 2020; https://gisdata2016-11-18t150447874z-azwater.opendata.arcgis.com/datasets/surface-watershed/explore?location=34.158174%2C-111.970823%2C7.50 Arizona: Arizona Ground Water Basins; Collected February, 2020; https://gisdata2016-11-18t150447874z-azwater.opendata.arcgis.com/datasets/groundwater-basin-2/explore?location=34.158174%2C-111.970823%2C7.50 California: California CalWater 2.2.1; Collected February, 2020; https://www.mlml.calstate.edu/mpsl-mlml/data-center/data-entry-tools/data-tools/gis-shapefile-layers/ Colorado: Colorado Water District Boundaries; Collected February, 2020; https://www.colorado.gov/pacific/cdss/gis-data-category Idaho: Idaho Department of Water Resources (IDWR) Administrative Basins; Collected November, 2015; https://data-idwr.opendata.arcgis.com/datasets/fb0df7d688a04074bad92ca8ef74cc26_4/explore?location=45.018686%2C-113.862284%2C6.93 Montana: Collected June, 2019; Directly contacted Montana Department of Natural Resources and Conservation (DNRC) Office of Information Technology (OIT) Nevada: Nevada State Engineer Admin Basin Boundaries; Collected April, 2020 https://ndwr.maps.arcgis.com/apps/mapviewer/index.html?layers=1364d0c3a0284fa1bcd90f952b2b9f1c New Mexico: New Mexico Office of the State Engineer (OSE) Declared Groundwater Basins; Collected April, 2020 https://geospatialdata-ose.opendata.arcgis.com/datasets/ose-declared-groundwater-basins/explore?location=34.179783%2C-105.996542%2C7.51 Oregon: Oregon Water Resources Department (OWRD) Administrative Basins; Collected February, 2020; https://www.oregon.gov/OWRD/access_Data/Pages/Data.aspx Utah: Utah Adjudication Books; Collected April, 2020; https://opendata.gis.utah.gov/datasets/utahDNR::utah-adjudication-books/explore?location=39.497165%2C-111.587782%2C-1.00 Washington: Washington Water Resource Inventory Areas (WRIA); Collected June, 2017; https://ecology.wa.gov/Research-Data/Data-resources/Geographic-Information-Systems-GIS/Data Wyoming: Wyoming State Engineer's Office Board of Control Water Districts; Collected June, 2019; Directly contacted Wyoming State Engineer's Office

Economics↗
DOE OSTI2021

Data Science and Computation for Rapid and Dynamic Compression Experiment Workflows at Experimental Facilities (Workshop Overview) [Slides]

Motivation for this workshop stems from these items: Compression experiments are important to LANL's core mission. The scientific community is sometimes frustrated at the pace of discovery. Data analytics for other experimental regimes are advancing. Now is the time as facilities are upgraded and coming online. Advances in data analytics and computational resources could push compression experiment discoveries to a new level.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

A High-Quality Workflow for Multi-Resolution Scientific Data Reduction and Visualization

Multi-resolution methods such as Adaptive Mesh Refinement (AMR) can enhance storage efficiency for HPC applications generating vast volumes of data. However, their applicability is limited and cannot be universally deployed across all applications. Furthermore, integrating lossy compression with multi-resolution techniques to further boost storage efficiency encounters significant barriers. To this end, we introduce an innovative workflow that facilitates high-quality multi-resolution data compression for both uniform and AMR simulations. Initially, to extend the usability of multi-resolution techniques, our workflow employs a compression-oriented Region of Interest (ROI) extraction method, transforming uniform data into a multi-resolution format. Subsequently, to bridge the gap between multi-resolution techniques and lossy compressors, we optimize three distinct compressors, ensuring their optimal performance on multi-resolution data. These optimizations can improve the compression ratio of SOTA approaches by up to 3.3× under the same data quality loss. Lastly, we incorporate an advanced uncertainty visualization method into our workflow to understand the potential impacts of lossy compression. Experimental evaluation demonstrates that our workflow achieves significant compression quality improvements.

Wang, Daoce↗