Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Data Compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,261 records · Page 70

The Athena Microscopic Imager Investigation

The Athena science payload on the Mars Exploration Rovers (MER) includes the Microscopic Imager (MI) [1]. The MI is a fixed-focus camera mounted on the end of an extendable instrument arm, the Instrument Deployment Device (IDD; see Figure 1).The MI was designed to acquire images at a spatial resolution of 30 microns/pixel over a broad spectral range (400 - 700 nm; see Table 1). Technically, the microscopic imager is not a microscope: it has a fixed magnification of 0.4 and is intended to produce images that simulate a geologist s view through a common hand lens. In photographers parlance, the system makes use of a macro lens. The MI uses the same electronics design as the other MER cameras [2, 3] but has optics that yield a field of view of 31 31 mm across a 1024 1024 pixel CCD image (Figure 2). The MI acquires images using only solar or skylightillumination of the target surface. A contact sensor is used to place the MI slightly closer to the target surface than its best focus distance (about 66 mm), allowing concave surfaces to be imaged in good focus. Because the MI has a relatively small depth of field (3 mm), a single MI image of a rough surface will contain both focused and unfocused areas. Coarse focusing will be achieved by moving the IDD away from a rock target after the contact sensor is activated. Multiple images taken at various distances will be acquired to ensure good focus on all parts of rough surfaces. By combining a set of images acquired in this way, a completely focused image can be assembled. Stereoscopic observations can be obtained by moving the MI laterally relative to its boresight. Estimates of the position and orientation of the MI for each acquired image will be stored in the rover computer and returned to Earth with the image data. The MI optics will be protected from the Martian environment by a retractable dust cover. The dust cover includes a Kapton window that is tinted orange to restrict the spectral bandpass to 500-700 nm, allowing color information to be obtained by taking images with the dust cover open and closed. The MI will image the same materials measured by other Athena instruments (including surfaces prepared by the Rock Abrasion Tool), as well as rock and soil targets of opportunity. Subsets of the full image array can be selected and/or pixels can be binned to reduce data volume. Image compression will be used to maximize the information contained in the data returned to Earth. The resulting MI data will place other MER instrument data in context and aid in petrologic and geologic interpretations of rocks and soils on Mars.

K E Herkenhoff↗

Active Thermography Based on Tensor Rank Decomposition

Principal Component Thermography applies Singular Value Decomposition (SVD) to post-process data that are derived from active thermographic inspections. SVD provides useful compression of the data and allows for better understanding of substructure and indications of potential damage. In the standard approach, SVD is applied to a certain reshaping of a three-dimensional data stack into a two-dimensional array. This work applies the CANDECOMP-PARAFAC (CP) tensor rank decomposition directly to the three-dimensional data to avoid the initial reshaping step in order to begin to develop an inspection method that can more accurately detect defects in non-homogeneous and anisotropic materials. Tests against simulated data that compare the CP decomposition method with traditional Principal Component Thermography based on SVD are described. Finally, the method of Proper Generalized Decomposition (PGD) is used to derive the CP decomposition, and its performance against other algorithms is also discussed.

Thermography↗

Compression of regions in the global advanced very high resolution radiometer 1-km data set

The global advanced very high resolution radiometer (AVHRR) 1-km data set is a 10-band image produced at USGS' EROS Data Center for the study of the world's land surfaces. The image contains masked regions for non-land areas which are identical in each band but vary between data sets. They comprise over 75 percent of this 9.7 gigabyte image. The mask is compressed once and stored separately from the land data which is compressed for each of the 10 bands. The mask is stored in a hierarchical format for multi-resolution decompression of geographic subwindows of the image. The land for each band is compressed by modifying a method that ignores fill values. This multi-spectral region compression efficiently compresses the region data and precludes fill values from interfering with land compression statistics. Results show that the masked regions in a one-byte test image (6.5 Gigabytes) compress to 0.2 percent of the 557,756,146 bytes they occupy in the original image, resulting in a compression ratio of 89.9 percent for the entire image.

Kess, Barbara L.↗

FPGA-based compression of streaming x-ray photon correlation spectroscopy data

A data acquisition system to perform real-time background subtraction and lower-level-discrimination-based compression of streaming x-ray photon correlation spectroscopy (XPCS) data from a fast charge-coupled device (CCD) area detector has been built and put into service at the Advanced Photon source (APS) at Argonne National Laboratory. A commercial frame grabber with on-board field-programmable gate array (FPGA) was used in the design, and continuously processes 60 frames per second each consisting of 1,024 x 1,024 pixels with up to 64512 photon hits per frame.

43 PARTICLE ACCELERATORS↗

A checkpoint compression study for high-performance computing systems

As high-performance computing systems continue to increase in size and complexity, higher failure rates and increased overheads for checkpoint/restart (CR) protocols have raised concerns about the practical viability of CR protocols for future systems. Previously, compression has proven to be a viable approach for reducing checkpoint data volumes and, thereby, reducing CR protocol overhead leading to improved application performance. In this article, we further explore compression-based CR optimization by exploring its baseline performance and scaling properties, evaluating whether improved compression algorithms might lead to even better application performance and comparing checkpoint compression against and alongside other software- and hardware-based optimizations. Our results highlights are: (1) compression is a very viable CR optimization; (2) generic, text-based compression algorithms appear to perform near optimally for checkpoint data compression and faster compression algorithms will not lead to better application performance; (3) compression-based optimizations fare well against and alongside other software-based optimizations; and (4) while hardware-based optimizations outperform software-based ones, they are not as cost effective.

97 MATHEMATICS AND COMPUTING↗

Community detection in hypergraphs via mutual information maximization

Abstract The hypergraph community detection problem seeks to identify groups of related vertices in hypergraph data. We propose an information-theoretic hypergraph community detection algorithm which compresses the observed data in terms of community labels and community-edge intersections. This algorithm can also be viewed as maximum-likelihood inference in a degree-corrected microcanonical stochastic blockmodel. We perform the compression/inference step via simulated annealing. Unlike several recent algorithms based on canonical models, our microcanonical algorithm does not require inference of statistical parameters such as vertex degrees or pairwise group connection rates. Through synthetic experiments, we find that our algorithm succeeds down to recently-conjectured thresholds for sparse random hypergraphs. We also find competitive performance in cluster recovery tasks on several hypergraph data sets.

97 MATHEMATICS AND COMPUTING↗

Leaf spectral reflectance and transmittance, San Lorenzo, Panama, 2020

This data package contains full-range (350 – 2500 nm) leaf reflectance and transmittance spectra of leaves collected at the San Lorenzo forest canopy crane site in Panama (PA-SLZ) from January to March 2020. Around 1000 leaves were measured from 50 species and various leaf phenological stages. Leaves were sampled from both the top of the canopy and multiple heights within the canopy from 10 vertical profiles and for six core species, from the top of the canopy. This data package includes processed leaf spectra (*.csv), and raw data from the spectroradiometer (compressed as .zip). Metadata files include data descriptions (_dd.csv) for tabular data and sample information and a description of the field campaign protocol (PA-SLZ_2020_Protocol.pdf). In addition to the leaf spectral measurements included here, these samples were also used for measurement of leaf gas exchange, carbon and nitrogen content and leaf mass per unit leaf area (LMA) contained in related data packages.

54 ENVIRONMENTAL SCIENCES↗

Comprehensive, Multi-Source Cyber-Security Events Data Set

This data set represents 58 consecutive days of de-identified event data collected from five sources within Los Alamos National Laboratory’s corporate, internal computer network. The data sources include Windows-based authentication events from both individual computers and centralized Active Directory domain controller servers; process start and stop events from individual Windows computers; Domain Name Service (DNS) lookups as collected on internal DNS servers; network flow data as collected on at several key router locations; and a set of well-defined red teaming events that present bad behavior within the 58 days. In total, the data set is approximately 12 gigabytes compressed across the five data elements and presents 1,648,275,307 events in total for 12,425 users, 17,684 computers, and 62,974 processes. Specific users that are well known system related (SYSTEM, Local Service) were not de-identified though any well-known administrators account were still de-identified. In the network flow data, well-known ports (e.g. 80, 443, etc) were not de-identified. All other users, computers, process, ports, times, and other details were de-identified as a unified set across all the data elements (e.g. U1 is the same U1 in all of the data). The specific timeframe used is not disclosed for security purposes. In addition, no data that allows association outside of LANL’s network is included. All data starts with a time epoch of 1 using a time resolution of 1 second. In the authentication data, failed authentication events are only included for users that had a successful authentication event somewhere within the data set.

97 MATHEMATICS AND COMPUTING↗

User's guide to the Nimbus-4 backscatter ultraviolet experiment data sets

The first year's data from the Nimbus 4 backscatter ultraviolet (BUV) experiment have been archived in the National Space Science Data Center (NSSDC). Backscattered radiances in the ultraviolet measured by the satellite were used to compute the global total ozone for the period April 1970 - April 1971. The data sets now in the NSSDC are the results obtained by the Ozone Processing Team, which has processed the data with the purpose of determining the best quality of the data. There are four basic sets of data available in the NSSDC representing various stages in processing. The primary data base contains organized and cleaned data in telemetry units. The radiance data has had most of the engineering calibrations performed. The detailed total ozone data is the result of computations to obtain the total ozone; the Compressed Total Ozone data is a convenient condensation of the detailed total ozone. Product data sets are also included.

Lowrey, B. E.↗

A multiprocessing architecture for real-time monitoring

A multitasking architecture for performing real-time monitoring and analysis using knowledge-based problem solving techniques is described. To handle asynchronous inputs and perform in real time, the system consists of three or more distributed processes which run concurrently and communicate via a message passing scheme. The Data Management Process acquires, compresses, and routes the incoming sensor data to other processes. The Inference Process consists of a high performance inference engine that performs a real-time analysis on the state and health of the physical system. The I/O Process receives sensor data from the Data Management Process and status messages and recommendations from the Inference Process, updates its graphical displays in real time, and acts as the interface to the console operator. The distributed architecture has been interfaced to an actual spacecraft (NASA's Hubble Space Telescope) and is able to process the incoming telemetry in real-time (i.e., several hundred data changes per second). The system is being used in two locations for different purposes: (1) in Sunnyville, California at the Space Telescope Test Control Center it is used in the preflight testing of the vehicle; and (2) in Greenbelt, Maryland at NASA/Goddard it is being used on an experimental basis in flight operations for health and safety monitoring.

Schmidt, James L.↗

Real-time analysis of telemetry data

This paper descibes a knowledge-based system for performing real-time monitoring and analysis of telemetry data from the NASA Hubble Space Telescope (HST). In order to handle asynchronous inputs and perform in real time the system consists of three or more separate processes, which run concurrently and communicate via a message passing scheme. The data management process gathers, compresses, and scales the incoming telemetry data befoe sending it to the other tasks. The inferencing process uses the incoming data to perform a real-time analysis of the state and health of the Space Telescope. The I/O process receives telemetry monitors from the data management process, updates its graphical displays in real time, and acts as the interface to the console operator. The three processes may run on the same or different computers. This system is currently under development and is being used to monitor testcases produced by the Bass Telemetry System in the Hardware/Software Integration Facility at Lockheed Missile and Space Co. in Sunnyvale, California.

Kao, Simon A.↗

Data on Material Properties and Panel Compressive Strength of a Plastic-bonded Material of Glass Cloth and Canvas

Results are presented of tests for determining the tensile, compressive, and bending properties of a material of plastic-bonding glass cloth and canvas layers. In addition, 10 panel specimens were tested in compression. Although the material is not satisfactory for primary structural use in aircraft when compared on a strength-weight basis with other materials in common use, there appears to be potential strength in the material that will require research for development. These points are considered in some detail in the concluding discussion of the report. An appendix shows that a higher tensile strength can be obtained by changes in the type of weave used in the glass-cloth reinforcement.

MATERIALS, PLASTIC-STRENGTH↗

BOREAS RSS-16 AIRSAR CM Images: Integrated Processor Version 6.1 Level-3b

The BOREAS RSS-16 team used satellite and aircraft SAR data in conjunction with various ground measurements to determine the moisture regime of the boreal forest. RSS-16 assisted with the acquisition and ordering of NASA JPL AIRSAR data collected from the NASA DC-8 aircraft. The NASA JPL AIRSAR is a side-looking imaging radar system that utilizes the SAR principle to obtain high resolution images that represent the radar backscatter of the imaged surface at different frequencies and polarizations. The information contained in each pixel of the AIRSAR data represents the radar backscatter for all possible combinations of horizontal and vertical transmit and receive polarizations (i.e., HH, HV, VH, and VV). Geographically, the data cover portions of the BOREAS SSA and NSA. Temporally, the data were acquired from 12-Aug-1993 to 31-Jul-1995. The level-3b AIRSAR CM data are in compressed Stokes matrix format, which has 10 bytes per pixel. From this data format, it is possible to synthesize a number of different radar backscatter measurements. The data are stored in binary image-format files. The data files are available on a CD-ROM (see document number 20010000884), or from the Oak Ridge National Laboratory (ORNL) Distributed Active Archive Center (DAAC).

Hall, Forrest G.↗

Reliable Transport over SpaceWire for James Webb Space Telescope (JWST) Focal Plane Electronics (FPE) Network

NASA's James Webb Space Telescope (JWST) faces difficult technical and budgetary challenges to overcome before it is scheduled launch in 2010. The Integrated Science Instrument Module (ISIM), shares these challenges. The major challenge addressed in this paper is the data network used to collect, process, compresses and store Infrared data. A total of 114 Mbps of raw information must be collected from 19 sources and delivered to the two redundant data processing units across a twenty meter deployed thermally restricted interface. Further data must be transferred to the solid-state recorder and the spacecraft. The JWST detectors are kept at cryogenic temperatures to obtain the sensitivity necessary to measure faint energy sources. The Focal Plane Electronics (FPE) that sample the detector, generate packets from the samples, and transmit these packets to the processing electronics must dissipate little power in order to help keep the detectors at these cold temperatures. Separating the low powered front-end electronics from the higher-powered processing electronics, and using a simple high-speed protocol to transmit the detector data minimize the power dissipation near the detectors. Low Voltage Differential Signaling (LVDS) drivers were considered an obvious choice for physical layer because of their high speed and low power. The mechanical restriction on the number cables across the thermal interface force the Image packets to be concentrated upon two high-speed links. These links connect the many image packet sources, Focal Plane Electronics (FPE), located near the cryogenic detectors to the processing electronics on the spacecraft structure. From 12 to 10,000 seconds of raw data are processed to make up an image, various algorithms integrate the pixel data Loss of commands to configure the detectors as well as the loss of science data itself may cause inefficiency in the use of the telescope that are unacceptable given the high cost of the observatory. This combination of requirements necessitates a redundant, fault tolerant, high- speed, low mass, low power network with a low Bit error Rate(1E-9- 1E-12). The ISIM systems team performed many studies of the various network architectures that meeting these requirements. The architecture selected uses the Spacewire protocol, with the addition of a new transport and network layer added to implement end-to-end reliable transport. The network and reliable transport mechanism must be implemented in hardware because of the high average information rate and the restriction on the ability of the detectors to buffer data due to power and size restrictions. This network and transport mechanism was designed to be compatible with existing Spacewire links and routers so that existing equipment and designs may be leveraged upon. The transport layer specification is being coordinated with European Space Agency (ESA), Spacewire Working Group and the Consultative Committee for Space Data System (CCSDS) PlK Standard Onboard Interface (SOIF) panel, with the intent of developing a standard for reliable transport for Spacewire. Changes to the protocol presented are likely since negotiations are ongoing with these groups. A block of RTL VHDL that implements a multi-port Spacewire router with an external user interface will be developed and integrated with an existing Spacewire Link design. The external user interface will be the local interface that sources and sinks packets onto and off of the network (Figure 3). The external user interface implements the network and transport layer and handles acknowledgements and re-tries of packets for reliable transport over the network. Because the design is written in RTL, it may be ported to any technology but will initially be targeted to the new Actel Accelerator series (AX) part. Each link will run at 160 Mbps and the power will be about 0.165 Watt per link worst case in the Actel AX.

Rakow, Glenn↗

Usage of Data-Encoded Web Maps with Client Side Color Rendering for Combined Data Access, Visualization and Modeling Purposes

Current approaches to satellite observation data storage and distribution implement separate visualization and data access methodologies which often leads to the need in time consuming data ordering and coding for applications requiring both visual representation as well as data handling and modeling capabilities. We describe an approach we implemented for a data-encoded web map service based on storing numerical data within server map tiles and subsequent client side data manipulation and map color rendering. The approach relies on storing data using the lossless compression Portable Network Graphics (PNG) image data format which is natively supported by web-browsers allowing on-the-fly browser rendering and modification of the map tiles. The method is easy to implement using existing software libraries and has the advantage of easy client side map color modifications, as well as spatial subsetting with physical parameter range filtering. This method is demonstrated for the ASTER-GDEM elevation model and selected MODIS data products and represents an alternative to the currently used storage and data access methods. One additional benefit includes providing multiple levels of averaging due to the need in generating map tiles at varying resolutions for various map magnification levels. We suggest that such merged data and mapping approach may be a viable alternative to existing static storage and data access methods for a wide array of combined simulation, data access and visualization purposes.

Pliutau, Denis↗