Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 199 records · Page 11
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Machine Learning Techniques for Data Reduction of Climate Applications

Scientists conduct large-scale simulations to compute derived quantities-of-interest (QoI) from primary data. Often, QoI are linked to specific features, regions, or time intervals, such that data can be adaptively reduced without compromising the integrity of QoI. For many spatiotemporal applications, these QoI are binary in nature and represent presence or absence of a physical phenomenon. We present a pipelined compression approach that first uses neural-network-based techniques to derive regions where QoI are highly likely to be present. Then, we employ a Guaranteed Autoencoder (GAE) to compress data with differential error bounds. GAE uses QoI information to apply low-error compression to only these regions. This results in overall high compression ratios while still achieving downstream goals of simulation or data collections. Experimental results are presented for climate data generated from the E3SM Simulation model for downstream quantities such as tropical cyclone and atmospheric river detection and tracking. These results show that our approach is superior to comparable methods in the literature.

Li, Xiao [University of Florida]↗
DOE OSTI2020

Performance Optimization for Relative-Error-Bounded Lossy Compression on Scientific Data

Scientific simulations in high-performance computing (HPC) environments generate vast volume of data, which may cause a severe I/O bottleneck at runtime and a huge burden on storage space for postanalysis. Unlike traditional data reduction schemes such as deduplication or lossless compression, not only can error-controlled lossy compression significantly reduce the data size but it also holds the promise to satisfy user demand on error control. Pointwise relative error bounds (i.e., compression errors depends on the data values) are widely used by many scientific applications with lossy compression since error control can adapt to the error bound in the dataset automatically. Pointwise relative-error-bounded compression is complicated and time consuming. In this article, we develop efficient precomputation-based mechanisms based on the SZ lossy compression framework. Our mechanisms can avoid costly logarithmic transformation and identify quantization factor values via a fast table lookup, greatly accelerating the relative-error-bounded compression with excellent compression ratios. In addition, we reduce traversing operations for Huffman decoding, significantly accelerating the decompression process in SZ. Experiments with eight well-known real-world scientific simulation datasets show that our solution can improve the compression and decompression rates (i.e., the speed) by about 40 and 80 p, respectively, in most of cases, making our designed lossy compression strategy the best-in-class solution in most cases.

96 KNOWLEDGE MANAGEMENT AND PRESERVATION↗
DOE OSTI2018

Efficient Lossy Compression for Scientific Data Based on Pointwise Relative Error Bound

An effective data compressor is becoming increasingly critical to today's scientific research, and many lossy compressors are developed in the context of absolute error bounds. Based on physical/chemical definitions of simulation fields or multiresolution demand, however, many scientific applications need to compress the data with a pointwise relative error bound (i.e., the smaller the data value, the smaller the compression error to tolerate). To this end, we propose two optimized lossy compression strategies under a state-of-the-art three-staged compression framework (prediction + quantization + entropy-encoding). The first strategy (called block-based strategy) splits the data set into many small blocks and computes an absolute error bound for each block, so it is particularly suitable for the data with relatively high consecutiveness in space. The second strategy (called multi-threshold-based strategy) splits the whole value range into multiple groups with exponentially increasing thresholds and performs the compression in each group separately, which is particularly suitable for the data with a relatively large value range and spiky value changes. We implement the two strategies rigorously and evaluate them comprehensively by using two scientific applications which both require lossy compression with point-wise relative error bound. Experiments show that the two strategies exhibit the best compression qualities on different types of data sets respectively. In conclusion, the compression ratio of our lossy compressor is higher than that of other state-of-the-art compressors by 17.2-618 percent on the climate simulation data and 30-210 percent on the N-body simulation data, with the same relative error bound and without degradation of the overall visualization effect of the entire data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Shock compression of poly(methyl methacrylate) PMMA in the 1000 GPa regime: Z machine experiments

Hydrocarbon polymers are used in a wide variety of practical applications. In the field of dynamic compression at extreme pressures, these polymers are used at several high energy density (HED) experimental facilities. One of the most common polymers is poly(methyl methacrylate) or PMMA, also called Plexiglass® or Lucite®. Here, we present high-fidelity, hundreds of GPa range experimental shock compression data measured on Sandia’s Z machine. We extend the principal shock Hugoniot for PMMA to more than threefold compression up to 650 GPa and re-shock Hugoniot states up to 1020 GPa in an off-Hugoniot regime, where experimental data are even sparser. These data can be used to put additional constraints on tabular equation of state (EOS) models. The present results provide clear evidence for the need to re-examine the existing tabular EOS models for PMMA above ~120 GPa as well as perhaps revisit EOSs of similar hydrocarbon polymers commonly used in HED experiments investigating dynamic compression, hydrodynamics, or inertial confinement fusion.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2021

Kinetic and temperature sensitivity properties of soil exoenzymes through the soil profile down to one-meter depth at a temperate coniferous forest (Blodgett, CA)

This dataset contains data on kinetic and temperature sensitivity parameters of the exoenzymes β-glucosidase (BG), leucine/leucyl aminopeptidase (LAP) and acid phosphatase (AP) across a soil profile down to 90 cm depth at Blodgett forest, CA, USA. These data were generated to determine if kinetic and thermal properties of microbial exoenzymes involved in organic matter decomposition varied with soil depth, following variation in soil properties and microbial communities, as part of the study: Alves et al. (2021). Kinetic Properties of Microbial Exoenzymes Vary with Soil Depth but Have Similar Temperature Sensitivities Through the Soil Profile. Frontiers in Microbiology 12:3618. https://doi.org/10.3389/fmicb.2021.735282. This research was performed within the framework of the TES Belowground Biogeochemistry SFA project, in particular association with a long-term field warming experiment of the whole soil profile at Blodgett forest. Samples for this work were collected from locations representative of the field experimental plots. Potential enzyme activity rates were measured using laboratory fluorometric assays with soils collected at 0-10, 10-20, 30-40, 50-60, 60-70 and 80-90 cm deep in biological triplicates (i.e., three soil cores collected at different representative locations). Assays with each soil were conducted over a gradient of eight substrate concentrations per enzyme, and incubated at 4, 10, 16, 25, 35 or 50°C. Enzyme Michaelis-Menten kinetics were modeled over the eight substrate concentrations at each temperature, and the temperature sensitivity of the kinetic parameters was modeled over the six temperatures using linear Arrhenius/Q10 and non-linear Macromolecular Rate Theory (MMRT) models. The dataset includes the fully processed enzyme activity rate data used to model Michaelis-Menten kinetics, calculated kinetic and temperature sensitivity parameters, and basic soil and microbial biomass chemistry for each sample. All data is provided for each individual biological replicate, and kinetic and temperature sensitivity parameters are also provided as means of the biological replicates. The dataset also includes all raw measurement data and code used to parse, combine and perform the analyses described by Alves et al. (2021). For file descriptions, see the file-level metadata files: “enzymes_dataProcessed_flmd.xlsx” (processed data); “enzymes_dataRaw_flmd.xlsx” (compressed raw data and metadata); and “enzymes_code_flmd.xlsx” (compressed code). The experimental design, list of parameters measured, soil and microbial biomass chemistry data, and means of biological replicates for kinetic and temperature sensitivity parameters are also provided as human-readable tables in file “enzymeTraits_design_results_tables.xlsx”.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Extended dynamic mode decomposition for model reduction in fluid dynamics simulations

High computational cost and storage/memory requirements of fluid dynamics simulations constrain their usefulness as a predictive tool. Reduced-order models (ROMs) provide a viable solution to this challenge by extracting the key underlying dynamics of a complex system directly from data. We investigate the efficacy and robustness of an extended dynamic mode decomposition (xDMD) algorithm in constructing ROMs of three-dimensional cardiovascular computations. Focusing on the ROMs' accuracy in representation and interpolation, we relate these metrics to the truncation rank of singular value decomposition, which underpins xDMD and other approaches to ROM construction. Our key innovation is to relate the truncation rank to the singular values of the original flow problem. This result establishes a priori guidelines for the xDMD deployment and its likely success as a means of data compression and reconstruction of the system's dynamics from dominant spatiotemporal structures present in the data.

Mechanics↗
DOE OSTI2021

All Optical Neural Networks for Low Power Edge Computing

We developed a simplistic physics-based model of an all-optical neural network that mimics the encoder part of an autoencoder neural network for image compression. Our approach relies on the generation of a MATLAB-based model for both data compression and decompression and utilizes MATLAB's built-in autoencoder networks in combination with simple propagation of optical fields between layers constituting phase elements via Fourier transform. We optimize the phase elements using the particle swarm optimization technique and using our model, we demonstrate a compression ratio of 25% for 2828-pixel input images containing numeric digits from 0 to 9.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2019

Real-time data processing in the ALICE High Level Trigger at the LHC

At the Large Hadron Collider at CERN in Geneva, Switzerland, atomic nuclei are collided at ultra-relativistic energies. Many final-state particles are produced in each collision and their properties are measured by the ALICE detector. The detector signals induced by the produced particles are digitized leading to data rates that are in excess of 48 GB/s. The ALICE High Level Trigger (HLT) system pioneered the use of FPGA- and GPU-based algorithms to reconstruct charged-particle trajectories and reduce the data size in real time. The results of the reconstruction of the collision events, available online, are used for high level data quality and detector-performance monitoring and real-time time-dependent detector calibration. The online data compression techniques developed and used in the ALICE HLT have more than quadrupled the amount of data that can be stored for offline event processing.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2018

Studies of CSR and Microbunching at Jefferson Laboratory ERLs

One attractive feature of energy recovery linacs (ERLs) is they are source limited. However as beam brightness increases so too do the effects of coherent synchrotron radiation (CSR) and the microbunching instability. The Low Energy Recirculator Facility at Jefferson Laboratory provides a test bed to characterize aspects of CSR's effect on the beam by measuring the energy extraction via CSR as a function of bunch compression. Data was recorded with acceleration occurring on the rising part of the RF waveform while the full compression point was moved along the backleg of the machine and the response of the beam measured. Acceleration was moved to the falling part of the RF waveform and the experiment repeated. Initial start-to-end simulations using a 1D CSR model show good agreement with measurements. The experiment motivated the design of a modified Continuous Electron Beam Accelerator Facility-style arc with control of CSR and the microbunching gain. Insights gained from that study informed designs for recirculation arcs in an ERL-driven electron cooler for Jefferson Laboratory's Electron Ion Collider. Progress on the design and outstanding challenges of the cooler are discussed.

Tennant, C.↗
DOE OSTI2020

Global analysis peak fitting for chemical spectroscopy data

The present invention relates to methods for analyzing a chemical sample. For instance, the methods herein allow for global analysis of spectroscopy data in order to extract useful chemical properties from complicated multidimensional data. Such analysis can optionally employ data compression to further expedite computer-implemented computation. In particular, the methods herein provide global analysis of data matrices explained by both linear and non-linear terms.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Non-Gaussianity in the weak lensing correlation function likelihood – implications for cosmological parameter biases

ABSTRACT We study the significance of non-Gaussianity in the likelihood of weak lensing shear two-point correlation functions, detecting significantly non-zero skewness and kurtosis in 1D marginal distributions of shear two-point correlation functions in simulated weak lensing data. We examine the implications in the context of future surveys, in particular LSST, with derivations of how the non-Gaussianity scales with survey area. We show that there is no significant bias in 1D posteriors of Ωm and σ8 due to the non-Gaussian likelihood distributions of shear correlations functions using the mock data (100 deg2). We also present a systematic approach to constructing approximate multivariate likelihoods with 1D parametric functions by assuming independence or more flexible non-parametric multivariate methods after decorrelating the data points using principal component analysis (PCA). While the use of PCA does not modify the non-Gaussianity of the multivariate likelihood, we find empirically that the 1D marginal sampling distributions of the PCA components exhibit less skewness and kurtosis than the original shear correlation functions. Modelling the likelihood with marginal parametric functions based on the assumption of independence between PCA components thus gives a lower limit for the biases. We further demonstrate that the difference in cosmological parameter constraints between the multivariate Gaussian likelihood model and more complex non-Gaussian likelihood models would be even smaller for an LSST-like survey. In addition, the PCA approach automatically serves as a data compression method, enabling the retention of the majority of the cosmological information while reducing the dimensionality of the data vector by a factor of ∼5.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Multifacets of lossy compression for scientific data in the Joint-Laboratory of Extreme Scale Computing

The Joint Laboratory on Extreme-Scale Computing (JLESC) was initiated at the same time lossy compression for scientific data became an important topic for the scientific communities. The teams involved in the JLESC played and are still playing an important role in developing the research, techniques, methods, and technologies making lossy compression for scientific data a key tool for scientists and engineers. Here, in this paper, we present the evolution of lossy compression for scientific data from 2015, describing the situation before the JLESC started, the evolution of this discipline in the past 8 years (until 2023) through the prism of the JLESC collaborations on this topic and some of the remaining open research questions.

Compression for AI↗
DOE OSTI2024

Dark energy survey year 3 results: likelihood-free, simulation-based w CDM inference with neural compression of weak-lensing map statistics

We present simulation-based cosmological wcold dark matter (wCDM) inference using dark energy survey year 3 weak-lensing maps, via neural data compression of weak-lensing map summary statistics: power spectra, peak counts, and direct map-level compression/inference with convolutional neural networks (CNN). Using simulation-based inference, also known as likelihood-free or implicit inference, we use forward-modelled mock data to estimate posterior probability distributions of unknown parameters. This approach allows all statistical assumptions and uncertainties to be propagated through the forward-modelled mock data; these include sky masks, non-Gaussian shape noise, shape measurement bias, source galaxy clustering, photometric redshift uncertainty, intrinsic galaxy alignments, non-Gaussian density fields, neutrinos, and non-linear summary statistics. We include a series of tests to validate our inference results. This paper also describes the Gower Street simulation suite: 791 full-sky pkdgrav3 dark matter simulations, with cosmological model parameters sampled with a mixed active-learning strategy, from which we construct over 3000 mock dark energy survey lensing data sets. For wCDM inference, for which we allow –1 < w < –$\frac{1}{3}$⁠, our most constraining result uses power spectra combined with map-level (CNN) inference. Using gravitational lensing data only, this map-level combination gives Ω m = 0.283$^{+0.020}_{–0.027}$⁠, S 8 = 0.804$^{+0.025}_{–0.017⁠}$, and w < –0.80 (with a 68 per cent credible interval); compared to the power spectrum inference, this is more than a factor of two improvement in dark energy parameter (Ω⁠ DE , w⁠) precision.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

ZFS Interface For Accelerators

As data volume increases in HPC centers, storing data in reasonable amounts of space and time while maintaining data integrity and recoverability becomes ever more difficult. ZFS is a filesystem that provides these features, among many others, that make it attractive for usage in HPC environments. It has the ability to compress data, compute checksums, and compute redundancy codes within the same runtime, rather than compute each separately, without knowledge of the underlying structure of the filesystem. However, measurements have shown that compression on CPUs can be incredibly inefficient and significantly reduce the performance of ZFS. The ZFS Interface for Accelerators (Z.I.A.) was developed to provide an interface to route data to accelerators while being processed by ZFS, so that the ZFS infrastructure and features are maintained, while allowing for ZFS administrators to provide faster implementations of compression, or other features, to their instance of ZFS.

Lee, Jason↗
DOE OSTI2026

Ramp-release experiments for strength measurements: Strain-rate dependence

This paper presents an enhanced analysis method for investigating material properties at high strain rates, extending the capability of established experimental techniques to gain more information. The ramp-release method has been applied to many experiments reported at high (≈10 5 − 10 6 s −1 ) strain-rates. More recent data gathered at the National Ignition Facility (NIF) has enabled higher (≈10 8 s −1 ) strain-rates to be studied. Here, we present an initial application of ramp-release analysis to NIF ramp-compression data, illustrating both the opportunities and the practical challenges of extending these methods to laser-driven platforms. The higher strain-rates accessed at the NIF mean that there is more strain-rate enhancement to strength, and the experimental configuration means that this enhancement is more readily seen in the data. This is enabled by the capability of avoiding peak-compression attenuation through the sample thickness with a designed hold period made possible by the pulse-shaping capability of NIF. We propose that this combination of experimental conditions and an enhanced analysis method enables the strain-rate enhancement to strength to be studied, and potentially for this to inform physics models at smaller scales than the continuum.

36 MATERIALS SCIENCE↗
DOE OSTI2014

Equation of state of bcc-Mo by static volume compression to 410 GPa

Unit cell volumes of Mo and Pt have been measured simultaneously to ≈400 GPa by x-ray powder diffraction using a diamond anvil cell and synchrotron radiation source. The body-centered cubic (bcc) phase of Mo was found to be stable up to 410 GPa. The equation of state (EOS) of bcc-Mo was determined on the basis of Pt pressure scale. A fit of Vinet EOS to the volume compression data gave K{sub 0} = 262.3(4.6) GPa, K{sub 0}′ = 4.55(16) with one atmosphere atomic volume V{sub 0} = 31.155(24) A{sup 3}. The EOS was in good agreement with the previous ultrasonic data within pressure difference of 2.5%–3.3% in the multimegabar range, though the EOS of Mo proposed from a shock compression experiment gave lower pressure by 7.2%–11.3% than the present EOS. The agreement would suggest that the Pt pressure scale provides an accurate pressure value in an ultra-high pressure range.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗