Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 217 records · Page 12
All sourcesNASA NTRSDOE OSTI
DOE OSTI2023

CMS High Granularity Calorimeter ECON-D ASIC overview and radiation testing results

The Compact Muon Solenoid (CMS) Experiment’s High Granularity Calorimeter (HGCAL) upgrade replaces the CMS electromagnetic and hadronic endcap calorimeters in preparation for the high-rate and high-radiation environment of the High Luminosity LHC. To effectively use the over 6 million channels of this highly-segmented “imaging” calorimeter, CMS is pioneering very front-end data compression with the Endcap Concentrator (ECON) ASICs – the ECON-T for the trigger path and the ECON-D for the data path. These 65 nm CMOS ASICs are radiation tolerant (200 Mrad) and low-power (< 2.5 mW/channel). In June 2023, we received the first full-functionality prototype of the data path concentrator ASIC, the ECON-D-P1. This talk will present an overview of the ECON-D-P1, summarize functionality and system testing, and present results from both Total Ionizing Dose (TID) and Single Event Effect (SEE) testing campaigns completed in summer 2023, validating the ECON-D radiation tolerant performance.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2020

Power-oriented bus encoding for data transmission

Systems, apparatuses, and methods for reducing the toggle rates on buses are disclosed. A computing system includes a source which provides packets for transmission on a bus. The packet is compressed by a compression engine. The compressed data format of the packet includes locations (bit positions) referred to as holes which do not include valid data. A bus configuration module identifies the locations of the holes and replaces the holes with information from a previous packet transmitted earlier on the bus. The bus configuration module also determines a new transmission bus width for the packet for lowering the bus toggle rate on the bus during transmission.

42 ENGINEERING↗
DOE OSTI2017

Canopus: A Paradigm Shift Towards Elastic Extreme-Scale Data Analytics on HPC

Scientific simulations on high performance computing (HPC) platforms generate large quantities of data. To bridge the widening gap between compute and I/O, and enable data to be more efficiently stored and analyzed, simulation outputs need to be refactored, reduced, and appropriately mapped to storage tiers. However, a systematic solution to support these steps has been lacking on the current HPC software ecosystem. To that end, this paper develops Canopus, a progressive JPEGlike data management scheme for storing and analyzing big scientific data. It co-designs the data decimation, compression and data storage, taking the hardware characteristics of each storage tier into considerations. With reasonably low overhead, our approach refactors simulation data into a much smaller, reduced-accuracy base dataset, and a series of deltas that is used to augment the accuracy if needed. The base dataset and deltas are compressed and written to multiple storage tiers. Data saved on different tiers can then be selectively retrieved to restore the level of accuracy that satisfies data analytics. Thus, Canopus provides a paradigm shift towards elastic data analytics and enables end users to make trade-offs between analysis speed and accuracy on-the-fly. We evaluate the impact of Canopus on unstructured triangular meshes, a pervasive data model used by scientific modeling and simulations. In particular, we demonstrate the progressive data exploration of Canopus using the “blob detection” use case on the fusion simulation data.

Lu, Tao↗
DOE OSTI2024

Weak-form latent space dynamics identification

Recent work in data-driven modeling has demonstrated that a weak formulation of model equations enhances the noise robustness of a wide range of computational methods. In this paper, we demonstrate the power of the weak form to enhance the LaSDI (Latent Space Dynamics Identification) algorithm, a recently developed data-driven reduced order modeling technique. We introduce a weak form-based version WLaSDI (Weak-form Latent Space Dynamics Identification). WLaSDI first compresses data, then projects onto the test functions and learns the local latent space models. Notably, WLaSDI demonstrates significantly enhanced robustness to noise. With WLaSDI, the local latent space is obtained using weak-form equation learning techniques. Compared to the standard sparse identification of nonlinear dynamics (SINDy) used in LaSDI, the variance reduction of the weak form guarantees a robust and precise latent space recovery, hence allowing for a fast, robust, and accurate simulation. We demonstrate the efficacy of WLaSDI vs. LaSDI on several common benchmark examples including viscid and inviscid Burgers', radial advection, and heat conduction. For instance, in the case of 1D inviscid Burgers' simulations with the addition of up to 100% Gaussian white noise, the relative error remains consistently below 6% for WLaSDI, while it can exceed 10,000% for LaSDI. Similarly, for radial advection simulations, the relative errors stay below 15% for WLaSDI, in stark contrast to the potential errors of up to 10,000% with LaSDI. Moreover, speedups of several orders of magnitude can be obtained with WLaSDI. For example applying WLaSDI to 1D Burgers' yields a 140X speedup compared to the corresponding full order model.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2022

Volumetric Rendering on Wavelet-Based Adaptive Grid

Numerical modeling of physical phenomena frequently involves processes across a wide range of spatial and temporal scales. In the last two decades, the advancements in wavelet-based numerical methodologies to solve partial differential equations, combined with the unique properties of wavelet analysis to resolve localized structures of the solution on dynamically adaptive computational meshes, make it feasible to perform large-scale numerical simulations of a variety of physical systems on a dynamically adaptive computational mesh that changes both in space and time. Volumetric visualization of the solution is an essential part of scientific computing, yet the existing volumetric visualization techniques do not take full advantage of multi-resolution wavelet analysis and are not fully tailored for visualization of a compressed solution on the wavelet-based adaptive computational mesh. Our objective is to explore the alternatives for the visualization of time-dependent data on space-time varying adaptive mesh using volume rendering while capitalizing on the available sparse data representation. Two alternative formulations are explored. The first one is based on volumetric ray casting of multi-scale datasets in wavelet space. Rather than working with the wavelets at the finest possible resolution, a partial inverse wavelet transform is performed as a preprocessing step to obtain scaling functions on a uniform grid at a user-prescribed resolution. As a result, a solution in physical space is represented by a superposition of scaling functions on a coarse regular grid and wavelets on an adaptive mesh. An efficient and accurate ray casting algorithm is based just on these coarse scaling functions. Additional details are added during the ray tracing by taking an appropriate number of wavelets into account based on support overlap with the interpolation point, wavelet coefficient magnitude, and other characteristics, such as opacity accumulation (front to back ordering) and deviation from frontal viewing direction. The second approach is based on complementing of wavelet-based adaptive mesh to the traditional Adaptive Mesh Refinement (AMR) mesh. Both algorithms are illustrated and compared to the existing volume visualization software for Rayleigh-Benard thermal convection and electron density data sets in terms of rendering time and visual quality for different data compression of both wavelet-based and AMR adaptive meshes.

Vezolainen, Alexei V.↗
DOE OSTI2022

Bloom Filter framework for Web Archives

The software provides a framework to build a fast look up layer that reflects the holdings of an archive or database and operates between search/discovery system and disk storage. The software utilizes Bloom filter (BF) data structure. The discovery service powered by the Bloom filter layer comes with a very high level of accuracy yet space-saving, since BF compresses data to fit into RAM.

Balakireva, Lyudmila↗
DOE OSTI2021

A Tailored Convolutional Neural Network for Nonlinear Manifold Learning of Computational Physics Data Using Unstructured Spatial Discretizations

In this work, we propose a nonlinear manifold learning technique based on deep convolutional autoencoders that is appropriate for model order reduction of physical systems in complex geometries. Convolutional neural networks have proven to be highly advantageous for compressing data arising from systems demonstrating a slow-decaying Kolmogorov n-width. However, these networks are restricted to data on structured meshes. Unstructured meshes are often required for performing analyses of real systems with complex geometry. Our custom graph convolution operators based on the available differential operators for a given spatial discretization effectively extend the application space of deep convolutional autoencoders to systems with arbitrarily complex geometry that are typically discretized using unstructured meshes. We propose sets of convolution operators based on the spatial derivative operators for the underlying spatial discretization, making the method particularly well suited to data arising from the solution of partial differential equations. We demonstrate the method using examples from heat transfer and fluid mechanics and show better than an order of magnitude improvement in accuracy over linear methods.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

LUNA Condition-Based Monitoring Update: Auto-encoders on Actuator Data [Slides]

An auto-encoder is a neural network trained to output a reconstruction of its input. There is a ‘bottleneck’ layer whose dimension is typically lower than that of the input. Because the input passes through the bottleneck, the auto-encoder must learn to compress (encode) the input into this low-dimensional form, and to decompress (decode) the compressed data into a close approximation of its original form.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2026

QProR: An Efficient Framework for Quantity-of-Interest Based Progressive Retrieval with Guaranteed Error Control

Scientific applications generate an unprecedented volume of data, overwhelming the network and file systems’ bandwidth and posing challenges for efficient and scalable data retrieval and analysis. Progressive data compression offers a promising solution by enabling on-demand retrieval at reduced size. However, existing progressive methods either fail to bound the errors in essential quantities of interest (QoIs) derived from raw data or suffer from suboptimal retrieval efficiency. In this work, we propose QProR, an efficient QoI-based progressive framework that optimizes progressive retrieval for target QoIs. Our key contributions include: (1) a systematic framework that integrates error-controlled lossy compressors with bitplane encoding while decoupling the two processes for high flexibility and adaptability; (2) a novel weighted bitplane encoding method which incorperates QoI knowledge into data refactoring to enhance retrieval efficiency; (3) an optimized retrieval strategy that accounts for the varying impacts of different variables on multivariate QoIs; (4) comprehensive evaluations using six real-world datasets from multiple scientific applications and thorough comparisons against state of the arts. Experimental results demonstrate that QProR achieves up to 80.38% reduction in the retrieval size under the same requested QoI error tolerance, when compared with the best-performing existing methods. When transferring 384 GB of scientific data to remote sites, QProR delivers up to 1.68 × speedup in the end-to-end data transfer performance.

Li, Wenbo [University of Kentucky]↗
DOE OSTI2020

SAMPA Based Streaming Readout Data Acquisition Prototype

We have assembled a small-scale streaming data acquisition system based on the SAMPA front-end ASIC. The 32-channel SAMPA chip was designed for the high-luminosity upgrade of the ALICE Time Projection Chamber (TPC) detector at the CERN Large Hadron Collider. The goals of the prototype system are to determine if the SAMPA chip is appropriate for use in detector systems at Jefferson Lab, and to gain experience with the hardware and software required to deploy streaming data acquisition systems in nuclear physics experiments. The 800 channel system is composed of components used in the ALICE TPC data acquisition upgrade. Five front-end cards (FEC) support five SAMPA chips each. SAMPA data streams on an FEC are concentrated into two high-speed (4.48 Gb/s) serial data streams by a pair Gigabit Transceiver ASICs (GBTx). These ten streams (44.8 Gb/s) are transmitted from the FECs over fibers to a PCIe based readout unit. The FPGA engine on the readout unit compresses data for transmission to a server via 100 Gb ethernet. Components on the FECs are radiation tolerant. High data rates can be handled. The system is by design scalable and thus provides a functional prototype for high-rate streaming readout at Jefferson Lab and the future Electron Ion Collider. We have made fundamental measurements (noise, linearity, time resolution) on the SAMPA ASIC. We have also coupled the readout system to a small Gas Electron Multiplier (GEM) detector and have studied its response to cosmic rays. A beam test of the system is planned.

ABBOTT, David↗
DOE OSTI2025

STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data

Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality—even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7 × faster than SZ3.

Wang, Daoce [University of Nebraska, Omaha]↗
DOE OSTI2019

Developing a radiation-aging model for polymeric foam cushions

Motivation: A major component of the A&L constitutive modeling effort over the last couple of years has involved developing a thermal aging model for foam cushions. Through extensive studies of uniaxial compression data, we have established that the Ogden N = 2 Hyperfoam in conjunction with the Tobolsky two-network model [Ref. 4] provides an accurate theoretical framework to describe the results, both for the traditional (stochastic) as well as the AM (3D-printed) foam pads. The results display numerical accuracy over a wide range of compression levels, from low to moderate up to slightly over 90% of lock-up. Over the last few weeks we have been exploring radiation aging data on a specific 3D printed foam, i.e., SE1700 AM FCT foam, to see if the Ogden + Tobolsky model can explain such data as well, and if so, how the various model parameters change as a function of dosage. As this report shows, the answer is indeed yes. We find that the modulus of the second (i.e., induced network) increases nearly proportionally with the radiation dosage, with a smaller sub-linear decrease in the primary shear modulus of the original network. This is consistent with the interpretation that exposure to energetic gamma-radiation is dominated by the formation of new cross-links. Significance: An important goal of A&L modeling efforts is to develop a fully age-aware materials model that can describe the joint effects of commonly present aging drivers like temperature, radiation, humidity, etc. Such a model is necessary to assess the lifetime of essential components like polymeric foams and rubber pads.

36 MATERIALS SCIENCE↗
DOE OSTI2025

A comparison between ShapeFit compression and Full-Modelling method with PyBird for DESI 2024 and beyond

DESI aims to provide one of the tightest constraints on cosmological parameters by analysing the clustering of more than thirty million galaxies. However, obtaining such constraints requires special care in validating the methodology and efforts to reduce the computational time required through data compression and emulation techniques. In this work, we perform a rigorous validation of the PyBird power spectrum modelling code with both a traditional emulated Full-Modelling approach and the model-independent ShapeFit compression approach. By using cubic box simulations that accurately reproduce the clustering and precision of the DESI survey, we find that the cosmological constraints from ShapeFit and Full-Modelling are consistent with each other at the ∼ 0.5σ level for the ΛCDM model. Both ShapeFit and Full-Modelling are also consistent with the true ΛCDM simulation cosmology down to a scale of k max = 0.20 hMpc -1 even after including the hexadecapole. For extended models such as the wCDM and the oCDM models, we find that including the hexadecapole can significantly improve the constraints and reduce the modelling errors with the same k max . While their discrepancies between the constraints from ShapeFit and Full-Modelling are more significant than ΛCDM, they remain consistent within 0.7σ. Lastly, we also show that the constraints on cosmological parameters with the correlation function evaluated from PyBird down to s min = 30h -1 Mpc are unbiased and consistent with the constraints from the power spectrum.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2022

Reduced order models for thermal radiative transfer problems based on moment equations and data-driven approximations of the Eddington tensor

Here a new group of structure and asymptotic preserving reduced-order models (ROMs) for multidimensional nonlinear thermal radiative transfer (TRT) problems is presented. They are formulated by means of the nonlinear projective approach and data compression techniques. The nonlinear projection is applied to the Boltzmann transport equation (BTE) to derive a hierarchy of low-order moment equations. Approximation of the Eddington tensor that provides exact closure for the system of moment equations is found with projection-based data-driven methodologies. These include the (i) proper orthogonal decomposition (POD), (ii) dynamic mode decomposition (DMD) and (iii) a variant of the DMD. A parameterization is derived for this ROM for the temperature of radiation incoming to the problem domain (the radiation drive temperature). This parameterization is informed from results of a dimensionless study of the TRT problem. Analysis of the ROMs is performed on the classical Fleck-Cummings TRT multigroup test problem in 2D geometry with a radiation-driven Marshak wave. Numerical results are presented to demonstrate the performance of these ROMs for the simulation of evolving radiation and heat waves. Results show these models to be sufficiently accurate for practical computations with rather low-rank representations of the Eddington tensor. As the rank of the approximation is increased, the errors of solutions generated by the ROMs gradually decreases.

42 ENGINEERING↗
DOE OSTI2021

A Preprocessing Tool for Enhanced Ion Mobility–Mass Spectrometry-Based Omics Workflows

The ability to improve the data quality of ion mobility–mass spectrometry (IM-MS) measurements is of great importance for enabling modular and efficient computational workflows and gaining better qualitative and quantitative insights from complex biological and environmental samples. We developed the PNNL PreProcessor, a standalone and user-friendly software housing various algorithmic implementations to generate new MS-files with enhanced signal quality and in the same instrument format. Different experimental approaches are supported for IM-MS based on Drift-Tube (DT) and Structures for Lossless Ion Manipulations (SLIM), including liquid chromatography (LC) and infusion analyses. The algorithms extend the dynamic range of the detection system, while reducing file sizes for faster and memory-efficient downstream processing. Specifically, multidimensional smoothing improves peak shapes of poorly defined low-abundance signals, and saturation repair reconstructs the intensity profile of high-abundance peaks from various analyte types. Further, other functionalities are data compression and interpolation, IM demultiplexing, noise filtering by low intensity threshold and spike removal, and exporting of acquisition metadata. Several advantages of the tool are illustrated, including an increase of 19.4% in lipid annotations and a two-times faster processing of LC-DT IM-MS data-independent acquisition spectra from a complex lipid extract of a standard human plasma sample. The software is freely available at https://omics.pnl.gov/software/pnnl-preprocessor.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2021

LANL-UW Collaboration on LC-QTOF Datasets (Annual Report)

Prior to the application of non-targeted chemometric techniques (i.e., Fisher ratio analysis), pre-processing steps must be carried out on the data collected using liquid chromatography-quadrupole time of flight (LC-QTOF) mass spectrometry. Data compression is the most important pre-processing step due to the vast number of high-resolution mass channels (m/z; 0.0001 Da) collected. Our first approach was to bin the mass spectral dimension to 0.01 Da; however, this approach was computationally expensive and did not preserve the high-resolution data.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Real-time data reduction at 100 Tbps: Challenge and opportunity for AI-based data reduction for next-generation large-scale nuclear physics collider experiment

The modern large-scale nuclear physics (NP) experiments in high-energy particle colliders utilize streaming-readout electronics to digitize detector response at O(100) Tbps bandwidth. Prominent examples at Brookhaven National Lab (BNL) include the sPHENIX experiment at Relativistic Heavy Ion Collider (RHIC), which is under construction, and the experiments proposed for the Electron-Ion Collider (EIC), planned for the 2030s . One of the main challenges for these streaming readout systems is to manage the data rate with sufficient data reduction in real time so the end-data fit persistent storage for offline analysis, which is typically at O(1000) times smaller and O(100) Gbps. Such data reduction traditionally is achieved via real-time high level triggers, which select and save a small subset of collisions of interest. Although triggering is applicable to high energy collider experiments such as those at the Large Hardron Collider at CERN, it is insufficient for these nuclear physics experiments which study diverse collision topologies. And traditional triggering approach is inefficient to preserve the max information harvested from the operation of colliders that costs O(100)M per year to DOE. Meanwhile, in recent years, ML-based high-throughput data reduction has emerged as a promising approach to efficiently preserve max information for a given space of persistent storage, e.g. via AI data compression, feature extraction, and noise filtering.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗