Results of adaptive predictor studies, part 1
Adaptative predictor studies based on Tiros cloud picture video data, and relative merits of linear and Markov prediction data compression techniques
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Adaptative predictor studies based on Tiros cloud picture video data, and relative merits of linear and Markov prediction data compression techniques
The Joint Laboratory on Extreme-Scale Computing (JLESC) was initiated at the same time lossy compression for scientific data became an important topic for the scientific communities. The teams involved in the JLESC played and are still playing an important role in developing the research, techniques, methods, and technologies making lossy compression for scientific data a key tool for scientists and engineers. Here, in this paper, we present the evolution of lossy compression for scientific data from 2015, describing the situation before the JLESC started, the evolution of this discipline in the past 8 years (until 2023) through the prism of the JLESC collaborations on this topic and some of the remaining open research questions.
We present simulation-based cosmological wcold dark matter (wCDM) inference using dark energy survey year 3 weak-lensing maps, via neural data compression of weak-lensing map summary statistics: power spectra, peak counts, and direct map-level compression/inference with convolutional neural networks (CNN). Using simulation-based inference, also known as likelihood-free or implicit inference, we use forward-modelled mock data to estimate posterior probability distributions of unknown parameters. This approach allows all statistical assumptions and uncertainties to be propagated through the forward-modelled mock data; these include sky masks, non-Gaussian shape noise, shape measurement bias, source galaxy clustering, photometric redshift uncertainty, intrinsic galaxy alignments, non-Gaussian density fields, neutrinos, and non-linear summary statistics. We include a series of tests to validate our inference results. This paper also describes the Gower Street simulation suite: 791 full-sky pkdgrav3 dark matter simulations, with cosmological model parameters sampled with a mixed active-learning strategy, from which we construct over 3000 mock dark energy survey lensing data sets. For wCDM inference, for which we allow –1 < w < –$\frac{1}{3}$, our most constraining result uses power spectra combined with map-level (CNN) inference. Using gravitational lensing data only, this map-level combination gives Ω m = 0.283$^{+0.020}_{–0.027}$, S 8 = 0.804$^{+0.025}_{–0.017}$, and w < –0.80 (with a 68 per cent credible interval); compared to the power spectrum inference, this is more than a factor of two improvement in dark energy parameter (Ω DE , w) precision.
Deformation within the North American plate in response to various tectonic processes is modeled using an elastic finite element analysis. The tectonic processes considered in the modeling include ridge forces associated with the normal thermal evolution of oceanic lithosphere, shear and normal stresses transmitted across transforms, normal stresses transmitted across convergent boundaries, stresses due to horizontal density contrasts within the continent, and shear tractions applied along the base of the plate. Model stresses are calculated with respect to a lithostatic reference stress state. Shear stresses transmitted across transform boundaries along the San Andreas and Caribbean are small, of the order of 5-10 MPa. Also, compressive stresses of the order of 5-10 MPa transmitted across the major transforms improve the fit to the data. Compressive stresses across convergent margins along the Aleutians and the Middle America trench are important.
The Aerospace Corporation developed the Electron PreProcessor (EPP) to support the Imaging Electron Spectrometer (IES) that is part of the RAPID experiment on the ESA/NASA CLUSTER mission. The purpose of the EPP is to collect raw data from the IES and perform processing and data compression on it before transferring it to the RAPID microprocessor system for formatting and transmission to the CLUSTER satellite data system. The report provides a short history of the RAPID and CLUSTER programs and describes the EPP design. Four EPP units were fabricated, tested, and delivered for the original CLUSTER program. These were destroyed during a launch failure. Four more EPP units were delivered for the CLUSTER II program. These were successfully launched and are operating nominally on orbit.
As data volume increases in HPC centers, storing data in reasonable amounts of space and time while maintaining data integrity and recoverability becomes ever more difficult. ZFS is a filesystem that provides these features, among many others, that make it attractive for usage in HPC environments. It has the ability to compress data, compute checksums, and compute redundancy codes within the same runtime, rather than compute each separately, without knowledge of the underlying structure of the filesystem. However, measurements have shown that compression on CPUs can be incredibly inefficient and significantly reduce the performance of ZFS. The ZFS Interface for Accelerators (Z.I.A.) was developed to provide an interface to route data to accelerators while being processed by ZFS, so that the ZFS infrastructure and features are maintained, while allowing for ZFS administrators to provide faster implementations of compression, or other features, to their instance of ZFS.
This paper presents an enhanced analysis method for investigating material properties at high strain rates, extending the capability of established experimental techniques to gain more information. The ramp-release method has been applied to many experiments reported at high (≈10 5 − 10 6 s −1 ) strain-rates. More recent data gathered at the National Ignition Facility (NIF) has enabled higher (≈10 8 s −1 ) strain-rates to be studied. Here, we present an initial application of ramp-release analysis to NIF ramp-compression data, illustrating both the opportunities and the practical challenges of extending these methods to laser-driven platforms. The higher strain-rates accessed at the NIF mean that there is more strain-rate enhancement to strength, and the experimental configuration means that this enhancement is more readily seen in the data. This is enabled by the capability of avoiding peak-compression attenuation through the sample thickness with a designed hold period made possible by the pulse-shaping capability of NIF. We propose that this combination of experimental conditions and an enhanced analysis method enables the strain-rate enhancement to strength to be studied, and potentially for this to inform physics models at smaller scales than the continuum.
Unit cell volumes of Mo and Pt have been measured simultaneously to ≈400 GPa by x-ray powder diffraction using a diamond anvil cell and synchrotron radiation source. The body-centered cubic (bcc) phase of Mo was found to be stable up to 410 GPa. The equation of state (EOS) of bcc-Mo was determined on the basis of Pt pressure scale. A fit of Vinet EOS to the volume compression data gave K{sub 0} = 262.3(4.6) GPa, K{sub 0}′ = 4.55(16) with one atmosphere atomic volume V{sub 0} = 31.155(24) A{sup 3}. The EOS was in good agreement with the previous ultrasonic data within pressure difference of 2.5%–3.3% in the multimegabar range, though the EOS of Mo proposed from a shock compression experiment gave lower pressure by 7.2%–11.3% than the present EOS. The agreement would suggest that the Pt pressure scale provides an accurate pressure value in an ultra-high pressure range.
A study of the energetics of solids leads to the conclusion that the equation of state for all classes of solids in compression can be expressed in terms of a universal function. The form of this universal function is determined by scaling experimental compression data for measured isotherms of a wide variety of solids. The equation of state is thus known (in the absence of phase transitions), if zero-pressure volume and isothermal compression and its pressure derivative are known. The discovery described in this paper has two immediate consequences: first, despite the well known differences in the microscopic energetics of the various classes of solids, there is a single equation of state for all classes in compression; and second, a new method is provided for analyzing measured isotherms and extrapolating high-pressure data from low-pressure (e.g. acoustic) data.
Recently, there has been a tremendous increase in the number of identified extrasolar planetary systems. Our understanding of their formation is tied to exoplanet internal structure models, which rely upon equations of state of light elements and compounds such as water. Here, we present shock compression data for water with unprecedented accuracy that show that water equations of state commonly used in planetary modeling significantly overestimate the compressibility at conditions relevant to planetary interiors. Furthermore, we show that its behavior at these conditions, including reflectivity and isentropic response, is well-described by a recent first-principles based equation of state. These findings advocate that this water model be used as the standard for modeling Neptune, Uranus, and “hot Neptune” exoplanets and should improve our understanding of these types of planets.
The primary objective of the NASA Earth-Sun Exploration Technology Office is to understand the observed Earth climate variability, thus enabling the determination and prediction of the climate's response to both natural and human-induced forcing. We are currently developing a suite of computational tools that will allow researchers to calculate, from data, a variety of information-theoretic quantities such as mutual information, which can be used to identify relationships among climate variables, and transfer entropy, which indicates the possibility of causal interactions. Our tools estimate these quantities along with their associated error bars, the latter of which is critical for describing the degree of uncertainty in the estimates. This work is based upon optimal binning techniques that we have developed for piecewise-constant, histogram-style models of the underlying density functions. Two useful side benefits have already been discovered. The first allows a researcher to determine whether there exist sufficient data to estimate the underlying probability density. The second permits one to determine an acceptable degree of round-off when compressing data for efficient transfer and storage. We also demonstrate how mutual information and transfer entropy can be applied so as to allow researchers not only to identify relations among climate variables, but also to characterize and quantify their possible causal interactions.
In previous papers we've shown how a well known data compression algorithm called Entropy-constrained Vector Quantization ( can be modified to reduce the size and complexity of very large, satellite data sets. In this paper, we descuss how to visualize and understand the content of such reduced data sets.
The Compact Muon Solenoid (CMS) Experiment’s High Granularity Calorimeter (HGCAL) upgrade replaces the CMS electromagnetic and hadronic endcap calorimeters in preparation for the high-rate and high-radiation environment of the High Luminosity LHC. To effectively use the over 6 million channels of this highly-segmented “imaging” calorimeter, CMS is pioneering very front-end data compression with the Endcap Concentrator (ECON) ASICs – the ECON-T for the trigger path and the ECON-D for the data path. These 65 nm CMOS ASICs are radiation tolerant (200 Mrad) and low-power (< 2.5 mW/channel). In June 2023, we received the first full-functionality prototype of the data path concentrator ASIC, the ECON-D-P1. This talk will present an overview of the ECON-D-P1, summarize functionality and system testing, and present results from both Total Ionizing Dose (TID) and Single Event Effect (SEE) testing campaigns completed in summer 2023, validating the ECON-D radiation tolerant performance.
Digital encoding/decoding system compresses color television image data in real time for transmission at lower data rates and, consequently, lower bandwidths. Implements predictive coding process, in which each picture element (pixel) predicted from values of prior neighboring pixels, and coded transmission expresses difference between actual and predicted current values. Combines differential pulse-code modulation process with non-linear, nonadaptive predictor, nonuniform quantizer, and multilevel Huffman encoder.
Systems, apparatuses, and methods for reducing the toggle rates on buses are disclosed. A computing system includes a source which provides packets for transmission on a bus. The packet is compressed by a compression engine. The compressed data format of the packet includes locations (bit positions) referred to as holes which do not include valid data. A bus configuration module identifies the locations of the holes and replaces the holes with information from a previous packet transmitted earlier on the bus. The bus configuration module also determines a new transmission bus width for the packet for lowering the bus toggle rate on the bus during transmission.
Scientific simulations on high performance computing (HPC) platforms generate large quantities of data. To bridge the widening gap between compute and I/O, and enable data to be more efficiently stored and analyzed, simulation outputs need to be refactored, reduced, and appropriately mapped to storage tiers. However, a systematic solution to support these steps has been lacking on the current HPC software ecosystem. To that end, this paper develops Canopus, a progressive JPEGlike data management scheme for storing and analyzing big scientific data. It co-designs the data decimation, compression and data storage, taking the hardware characteristics of each storage tier into considerations. With reasonably low overhead, our approach refactors simulation data into a much smaller, reduced-accuracy base dataset, and a series of deltas that is used to augment the accuracy if needed. The base dataset and deltas are compressed and written to multiple storage tiers. Data saved on different tiers can then be selectively retrieved to restore the level of accuracy that satisfies data analytics. Thus, Canopus provides a paradigm shift towards elastic data analytics and enables end users to make trade-offs between analysis speed and accuracy on-the-fly. We evaluate the impact of Canopus on unstructured triangular meshes, a pervasive data model used by scientific modeling and simulations. In particular, we demonstrate the progressive data exploration of Canopus using the “blob detection” use case on the fusion simulation data.
The fast KL transform algorithm was applied for data compression of a set of four ERTS multispectral images and its performance was compared with other techniques previously studied on the same image data. The performance criteria used here are mean square error and signal to noise ratio. The results obtained show a superior performance of the fast KL transform coding algorithm on the data set used with respect to the above stated perfomance criteria. A summary of the results is given in Chapter I and details of comparisons and discussion on conclusions are given in Chapter IV.
Lossless data compression was studied for many NASA missions. The Rice algorithm was demonstrated to provide better performance than other available techniques on most scientific data. A top-level description of the Rice algorithm is first given, along with some new capabilities implemented in both software and hardware forms. Systems issues important for onboard implementation, including sensor calibration, error propagation, and data packetization, are addressed. The latter part of the guide provides twelve case study examples drawn from a broad spectrum of science instruments.