Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 433 records · Page 24
All sourcesNASA NTRSDOE OSTI
DOE OSTI2015

A checkpoint compression study for high-performance computing systems

As high-performance computing systems continue to increase in size and complexity, higher failure rates and increased overheads for checkpoint/restart (CR) protocols have raised concerns about the practical viability of CR protocols for future systems. Previously, compression has proven to be a viable approach for reducing checkpoint data volumes and, thereby, reducing CR protocol overhead leading to improved application performance. In this article, we further explore compression-based CR optimization by exploring its baseline performance and scaling properties, evaluating whether improved compression algorithms might lead to even better application performance and comparing checkpoint compression against and alongside other software- and hardware-based optimizations. Our results highlights are: (1) compression is a very viable CR optimization; (2) generic, text-based compression algorithms appear to perform near optimally for checkpoint data compression and faster compression algorithms will not lead to better application performance; (3) compression-based optimizations fare well against and alongside other software-based optimizations; and (4) while hardware-based optimizations outperform software-based ones, they are not as cost effective.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Community detection in hypergraphs via mutual information maximization

Abstract The hypergraph community detection problem seeks to identify groups of related vertices in hypergraph data. We propose an information-theoretic hypergraph community detection algorithm which compresses the observed data in terms of community labels and community-edge intersections. This algorithm can also be viewed as maximum-likelihood inference in a degree-corrected microcanonical stochastic blockmodel. We perform the compression/inference step via simulated annealing. Unlike several recent algorithms based on canonical models, our microcanonical algorithm does not require inference of statistical parameters such as vertex degrees or pairwise group connection rates. Through synthetic experiments, we find that our algorithm succeeds down to recently-conjectured thresholds for sparse random hypergraphs. We also find competitive performance in cluster recovery tasks on several hypergraph data sets.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

Leaf spectral reflectance and transmittance, San Lorenzo, Panama, 2020

This data package contains full-range (350 – 2500 nm) leaf reflectance and transmittance spectra of leaves collected at the San Lorenzo forest canopy crane site in Panama (PA-SLZ) from January to March 2020. Around 1000 leaves were measured from 50 species and various leaf phenological stages. Leaves were sampled from both the top of the canopy and multiple heights within the canopy from 10 vertical profiles and for six core species, from the top of the canopy. This data package includes processed leaf spectra (*.csv), and raw data from the spectroradiometer (compressed as .zip). Metadata files include data descriptions (_dd.csv) for tabular data and sample information and a description of the field campaign protocol (PA-SLZ_2020_Protocol.pdf). In addition to the leaf spectral measurements included here, these samples were also used for measurement of leaf gas exchange, carbon and nitrogen content and leaf mass per unit leaf area (LMA) contained in related data packages.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2015

Comprehensive, Multi-Source Cyber-Security Events Data Set

This data set represents 58 consecutive days of de-identified event data collected from five sources within Los Alamos National Laboratory’s corporate, internal computer network. The data sources include Windows-based authentication events from both individual computers and centralized Active Directory domain controller servers; process start and stop events from individual Windows computers; Domain Name Service (DNS) lookups as collected on internal DNS servers; network flow data as collected on at several key router locations; and a set of well-defined red teaming events that present bad behavior within the 58 days. In total, the data set is approximately 12 gigabytes compressed across the five data elements and presents 1,648,275,307 events in total for 12,425 users, 17,684 computers, and 62,974 processes. Specific users that are well known system related (SYSTEM, Local Service) were not de-identified though any well-known administrators account were still de-identified. In the network flow data, well-known ports (e.g. 80, 443, etc) were not de-identified. All other users, computers, process, ports, times, and other details were de-identified as a unified set across all the data elements (e.g. U1 is the same U1 in all of the data). The specific timeframe used is not disclosed for security purposes. In addition, no data that allows association outside of LANL’s network is included. All data starts with a time epoch of 1 using a time resolution of 1 second. In the authentication data, failed authentication events are only included for users that had a successful authentication event somewhere within the data set.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

What to Support When You’re Compressing

Over the last nearly 20 years, lossy compression has become an essential aspect of HPC applications’ data pipelines, allowing them to overcome limitations in storage capacity and bandwidth and, in some cases, increase computational throughput and capacity. However, with the adoption of lossy compression comes the requirement to assess and control the impact lossy compression has on scientific outcomes. In this work, we take a major step forward in describing the state of practice and by characterizing workloads. We examine applications’ needs and compressors’ capabilities across 9 different supercomputing application domains. We present 24 takeaways that provide best practices for applications, operational impacts for facilities achieving compressed data, and gaps in application needs not addressed by production compressors that point towards opportunities for future compression research.

Error-Bounded Lossy Compression↗
DOE OSTI2021

Equation-of-state, sound speed, and reshock of shock-compressed fluid carbon dioxide

Mechanical equation-of-state data of initially liquid and solid CO2 shock-compressed to terapascal conditions are reported. Diamond-sapphire anvil cells were used to vary the initial density and state of CO2 samples that were then further compressed with laser-driven shock waves, resulting in a data set from which precise derivative quantities, including Grüneisen parameter and sound speed, are determined. Reshock states are measured to 800 GPa and map the same pressure-density conditions as the single shock using different thermodynamic paths. The compressibility data reported here do not support current density-functional-theory calculations, but are better represented by tabular equation-of-state models.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

StOKeDMD: Streaming Occupation kernel dynamic mode decomposition

Dynamic mode decomposition (DMD) has become a common technique for constructing surrogate models for dynamical systems from observed system states. The Occupation Kernel DMD (OKDMD) method proposed in (Rosenfeld et al., 2022) and (Rosenfeld et al., 2024) is a Liouville operator based method that builds surrogate models from system state trajectories. Here, this paper proposes an extension of OKDMD to the case when the system states are observed in a streaming fashion, i.e., only a small fraction of the state trajectory is available at a given time. The developed method, Streaming Occupation Kernel DMD (StOKeDMD), accommodates the streaming data input by leveraging properties of specific choices of kernel functions and occupation kernels. We apply the StoKeDMD method as a compression method for streaming data, analyze the memory complexity, and demonstrate the performance of StoKeDMD in the compression of streaming data generated from a Lorenz system and a fluid flow simulation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Optimising the processing and storage of visibilities using lossy compression

The next-generation radio astronomy instruments are providing a massive increase in sensitivity and coverage, largely through increasing the number of stations in the array and the frequency span sampled. The two primary problems encountered when processing the resultant avalanche of data are the need for abundant storage and the constraints imposed by I/O, as I/O bandwidths drop significantly on cold storage. An example of this is the data deluge expected from the SKA Telescopes of more than 60 PB per day, all to be stored on the buffer filesystem. While compressing the data is an obvious solution, the impacts on the final data products are hard to predict. In this paper, we chose an error-controlled compressor – MGARD – and applied it to simulated SKA-Mid and real pathfinder visibility data, in noise-free and noise-dominated regimes. As the data have an implicit error level in the system temperature, using an error bound in compression provides a natural metric for compression. MGARD ensures the compression incurred errors adhere to the user-prescribed tolerance. To measure the degradation of images reconstructed using the lossy compressed data, we proposed a list of diagnostic measures, exploring the trade-off between these error bounds and the corresponding compression ratios, as well as the impact on science quality derived from the lossy compressed data products through a series of experiments. We studied the global and local impacts on the output images for continuum and spectral line examples. We found relative error bounds of as much as 10%, which provide compression ratios of about 20, have a limited impact on the continuum imaging as the increased noise is less than the image RMS, whereas a 1% error bound (compression ratio of 8) introduces an increase in noise of about an order of magnitude less than the image RMS. For extremely sensitive observations and for very precious data, we would recommend a 0.1% error bound with compression ratios of about 4. These have noise impacts two orders of magnitude less than the image RMS levels. At these levels, the limits are due to instabilities in the deconvolution methods. We compared the results to the alternative compression tool DYSCO, in both the impacts on the images and in the relative flexibility. MGARD provides better compression for similar error bounds and has a host of potentially powerful additional features.

Techniques: interferometric↗
DOE OSTI2015

Hugoniot equation of state and dynamic strength of boron carbide

Boron carbide ceramics have been particularly problematic in attempts to develop adequate constitutive model descriptions for purposes of analysis of dynamic response in the shock and impact environment. Dynamic strength properties of boron carbide ceramic differ uniquely from comparable ceramics. Furthermore, boron carbide is suspected, but not definitely shown, to undergoing polymorphic phase transformation under shock compression. In the present paper, shock-wave compression measurements conducted over the past 40 years are assessed for the purpose of achieving improved understanding of the dynamic equation of state and strength of boron carbide. In particular, attention is focused on the often ignored Los Alamos National Laboratory (LANL) Hugoniot measurements performed on porous sintered boron carbide ceramic. The LANL data are shown to exhibit two compression anomalies on the shock Hugoniot within the range of 20–60 GPa that may relate to crystallographic structure transitions. More recent molecular dynamics simulations on the compressibility of the boron carbide crystal lattice reveal compression transitions that bear similarities to the LANL Hugoniot results. The same Hugoniot data are complemented with dynamic isentropic compression data for boron carbide extracted from Hugoniot measurements on boron carbide and copper granular mixtures. Other Hugoniot measurements, however, performed on near-full-density boron carbide ceramic differ markedly from the LANL Hugoniot data. These later data exhibit markedly less compressibility and tend not to show comparable anomalies in compressibility. Alternative Hugoniot anomalies, however, are exhibited by the near-full-density data. Experimental uncertainty, Hugoniot strength, and phase transformation physics are all possible explanations for the observed discrepancies. It is reasoned that experimental uncertainty and Hugoniot strength are not likely explanations for the observed differences. The notable mechanistic difference in the processes of shock compression between the LANL data and that of the other studies is the markedly larger inelastic deformation and dissipation experienced in the shock event brought about by compaction of the substantially larger porosity LANL test ceramics. High-pressure diamond anvil cell experiments reveal extensive amorphization, reasoned to be a reversion product of a higher-pressure crystallographic phase, which is a consequence of application of both high pressure and shear deformation to the boron carbide crystal structure. A dependence of shock-induced high-pressure phase transformation in boron carbide on the extent of shear deformation experienced in the shock process offers a plausible explanation for the differences observed in the LANL Hugoniot data on porous ceramic and that of other shock data on near-full-density boron carbide.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2023

HarDWR - Water Management Area (WMA) Shapefiles

For a detailed description of the database of which this record is only one part, please see the HarDWR meta-record. Borders of all Water Management Areas (WMAs) across the 11 Western-most states of the continental United States are available filtered through a single source. The legal name for this set of boundaries varies state-by-state. The data is provided as two compressed shapefiles. One, stateWMAs, contains data for all 11 states. For 10 of those states, Arizona being the exception, the polygons represent the legal management boundaries used by those states to manage their surface and groundwater resources respectively. Arizona is unique among this collection of states in that surface and groundwater resources are managed using two separate sets of boundaries. During our followup analysis, due to technical reasons we decided to focus on one set of boundaries, those for surface water. Due to this, the Arizona surface WMAs are included within stateWMAs. The Arizona groundwater WMAs are provided as the second file azGroundWMAs as a companion to the first file for completeness and general reference, although the boundaries were no longer referenced in the analysis (Grogan et al., in review) past a certain point. The columns for both shapefiles are: basinNum: the state provided unique numerical ID basinName: the state provided English name of the area, where applicable state: the state name uniID: a unique identifier we created by concatenating the state name, and underscore, and the state numerical ID

Economics↗
DOE OSTI2020

HarDWR - Water Management Area (WMA) Shapefiles

A dataset within the Harmonized Database of Western U.S. Water Rights (HarDWR). For a detailed description of the database, please see the meta-record v2.0. Changelog v2.0 - No changes v1.0 - Initial public release Description Borders of all Water Management Areas (WMAs) across the 11 western-most states of the coterminous United States are available filtered through a single source. The legal name for this set of boundaries varies state-by-state. The data is provided as two compressed shapefiles. One, stateWMAs, contains data for all 11 states. For 10 of those states, Arizona being the exception, the polygons represent the legal management boundaries used by those states to manage their surface and groundwater resources respectively. WMAs refer to the set of boundaries a particular state uses to manage its water resources. Each set of boundaries was collected from the states individually, and then merged into one spatial layer. The merging process included renaming some columns to enable merging with all other source layers, as well as removing columns deemed not required for followup analysis. The retained columns for each boundary are: basinNum - the state provided unique numerical ID; basinName - the state provided English name of the area, where applicable; state - the state name; and uniID - a unique identifier we created by concatenating the state name, and underscore, and the state numerical ID. Arizona is unique within this collection of states in that surface and groundwater resources are managed using two separate sets of boundaries. During our followup analysis (Grogan et al., in review) we decided to focus on one set of boundaries, those for surface water. This is due to the recommendation of our hydrologists that the surface water boundary set is a more realist representation of how water moves across the landscape, as a few of the groundwater boundaries are based on political and/or economic considerations. Therefore, the Arizona surface WMAs are included within stateWMAs. The Arizona groundwater WMAs are provided as a separate file, azGroundWMAs, as a companion to the first file for completeness and general reference. WMA spatial boundary data sources by state: Arizona: Arizona Surface Water Watersheds; Collected February, 2020; https://gisdata2016-11-18t150447874z-azwater.opendata.arcgis.com/datasets/surface-watershed/explore?location=34.158174%2C-111.970823%2C7.50 Arizona: Arizona Ground Water Basins; Collected February, 2020; https://gisdata2016-11-18t150447874z-azwater.opendata.arcgis.com/datasets/groundwater-basin-2/explore?location=34.158174%2C-111.970823%2C7.50 California: California CalWater 2.2.1; Collected February, 2020; https://www.mlml.calstate.edu/mpsl-mlml/data-center/data-entry-tools/data-tools/gis-shapefile-layers/ Colorado: Colorado Water District Boundaries; Collected February, 2020; https://www.colorado.gov/pacific/cdss/gis-data-category Idaho: Idaho Department of Water Resources (IDWR) Administrative Basins; Collected November, 2015; https://data-idwr.opendata.arcgis.com/datasets/fb0df7d688a04074bad92ca8ef74cc26_4/explore?location=45.018686%2C-113.862284%2C6.93 Montana: Collected June, 2019; Directly contacted Montana Department of Natural Resources and Conservation (DNRC) Office of Information Technology (OIT) Nevada: Nevada State Engineer Admin Basin Boundaries; Collected April, 2020 https://ndwr.maps.arcgis.com/apps/mapviewer/index.html?layers=1364d0c3a0284fa1bcd90f952b2b9f1c New Mexico: New Mexico Office of the State Engineer (OSE) Declared Groundwater Basins; Collected April, 2020 https://geospatialdata-ose.opendata.arcgis.com/datasets/ose-declared-groundwater-basins/explore?location=34.179783%2C-105.996542%2C7.51 Oregon: Oregon Water Resources Department (OWRD) Administrative Basins; Collected February, 2020; https://www.oregon.gov/OWRD/access_Data/Pages/Data.aspx Utah: Utah Adjudication Books; Collected April, 2020; https://opendata.gis.utah.gov/datasets/utahDNR::utah-adjudication-books/explore?location=39.497165%2C-111.587782%2C-1.00 Washington: Washington Water Resource Inventory Areas (WRIA); Collected June, 2017; https://ecology.wa.gov/Research-Data/Data-resources/Geographic-Information-Systems-GIS/Data Wyoming: Wyoming State Engineer's Office Board of Control Water Districts; Collected June, 2019; Directly contacted Wyoming State Engineer's Office

Economics↗
DOE OSTI2021

Data Science and Computation for Rapid and Dynamic Compression Experiment Workflows at Experimental Facilities (Workshop Overview) [Slides]

Motivation for this workshop stems from these items: Compression experiments are important to LANL's core mission. The scientific community is sometimes frustrated at the pace of discovery. Data analytics for other experimental regimes are advancing. Now is the time as facilities are upgraded and coming online. Advances in data analytics and computational resources could push compression experiment discoveries to a new level.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

A High-Quality Workflow for Multi-Resolution Scientific Data Reduction and Visualization

Multi-resolution methods such as Adaptive Mesh Refinement (AMR) can enhance storage efficiency for HPC applications generating vast volumes of data. However, their applicability is limited and cannot be universally deployed across all applications. Furthermore, integrating lossy compression with multi-resolution techniques to further boost storage efficiency encounters significant barriers. To this end, we introduce an innovative workflow that facilitates high-quality multi-resolution data compression for both uniform and AMR simulations. Initially, to extend the usability of multi-resolution techniques, our workflow employs a compression-oriented Region of Interest (ROI) extraction method, transforming uniform data into a multi-resolution format. Subsequently, to bridge the gap between multi-resolution techniques and lossy compressors, we optimize three distinct compressors, ensuring their optimal performance on multi-resolution data. These optimizations can improve the compression ratio of SOTA approaches by up to 3.3× under the same data quality loss. Lastly, we incorporate an advanced uncertainty visualization method into our workflow to understand the potential impacts of lossy compression. Experimental evaluation demonstrates that our workflow achieves significant compression quality improvements.

Wang, Daoce↗
DOE OSTI2015

Mock-Up activities for the extraction of samples from a salt-stone disposal unit - 15502

Mock-up testing with respect to the core-drilling of simulant salt-stone grout was conducted to support the future retrieval of field-emplaced salt-stone samples from salt-stone Disposal Unit (SDU) Cell 2A. The primary objectives of extracting samples from SDU Cell 2A are to enable measurement of field-emplaced samples with respect to properties that influence long-term performance, such as saturated hydraulic conductivity, and to compare the measured properties for field-emplaced and comparable samples prepared in the laboratory. Initial mock-up activities (Phases 1 and 2) were conducted in Fiscal Years (FY) 2013 and 2014 in order to demonstrate the viability of standard core-drilling techniques that are typically employed for sampling concrete structures. The primary requirement for the core-drilling technique was that it should minimize the potential of physical or chemical sample changes that would artificially degrade key grout properties. Mock-up activities were aimed at optimizing the techniques for core-drilling, and subsequent sample extraction, using a set-up that mimicked, to the extent practical, both the physical characteristics, and operational complexities, of core-drilling salt-stone grout from the roof of the SDU. Samples extracted from the grout monolith were subsequently characterized with respect to saturated hydraulic conductivity and compressive strength, and compared to data for cast samples prepared using the same simulant salt-stone material. The comparison of these properties for the two sample types was expected to highlight potential flaws (e.g., micro-cracking) introduced by the core-drilling process. While the data for saturated hydraulic conductivity were virtually identical (approximately 2 E-09 cm/sec), the compressive strength of the core-drilled samples (approximately 3.5 MPa (500 psi)) was on average half that measured for the cast samples. It was suspected that increased water saturation and surface scoring of the core-drilled samples might account for the disparate compressive Strength data. During Phases 1 and 2 it was determined that cores could be successfully drilled and extracted to a depth of approximately 0.9 meters (3 feet) below the grout surface. Phase 3 (FY 2105) has been initiated and is primarily directed at demonstrating the ability to core-drill through approximately 1.8 meters (6 feet) of overlying grout to retrieve the desired sample from the SDU. The original drill depth of 0.6 - 0.9 meters (2-3 feet) was based on an SDU fill height of 5.6 meters (18.5 feet) and sample location of approximately 4.9 meters (16 feet); recently the allowable fill height in the SDUs has been increased to 6.5 meters (21.25 feet); hence the increased depth of overlying grout above the samples to be retrieved. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗