Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data compression”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21
All sourcesNASA NTRSDOE OSTI
DOE OSTI2017

Optimization of SAMtools sorting using OpenMP tasks

SAMtools is a widely-used genomics application for post-processing high-throughput sequence alignment data. Such sequence alignment data are commonly sorted to make downstream analysis more efficient. However, this sorting process itself can be computationally- and I/O-intensive: high-throughput sequence alignment files in the de facto standard binary alignment/map (BAM) format can be many gigabytes in size, and may need to be decompressed before sorting and compressed afterwards. As a result, BAM-file sorting can be a bottleneck in genomics workflows. Here, this paper describes a case study on the performance analysis and optimization of SAMtools for sorting large BAM files. OpenMP task parallelism and memory optimization techniques resulted in a speedup of 5.9X versus the upstream SAMtools 1.3.1 for an internal (in-memory) sort of 24.6 GiB of compressed BAM data (102.6 GiB uncompressed) with 32 processor cores, while a 1.98X speedup was achieved for an external (out-of-core) sort of a 271.4 GiB BAM file.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Hugoniot Models for Na and LiF from LEOS

In this document, we provide the Hugoniot for sodium from two models: LEOS table L110 and Lynx table 110. We also provide the Hugoniot for lithium fluoride from LEOS (L2240) and Lynx (2240). The Hugoniot pressures are supplied for temperatures between 338.0 and 1.16×10 9 Kelvin and densities between 0.968 and 11.5 g/cc. These LEOS models were developed by the quotidian EOS methodology, which is a widely used and robust method for producing tabular EOS data. Tables list the model data for LEOS 110, Lynx 110, LEOS 2240, and Lynx 2240. The Lynx models follow the same methodology as the LEOS models; however, the Purgatorio average-atom DFT code was used to compute the electron thermal part of the EOS. The models for Lynx are only listed at high compression due to known issues with the Lynx library at lower pressures.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Custom NEON AOP reflectance mosaics and maps of shade masks, canopy water content

This mosaic of visible to shortwave infrared (VSWIR) data was derived from the assignable asset NEON AOP radiance data that was collected by LBNL’s Watershed Function SFA during the summer of 2018 (DOI: 10.15485/1617204). This atmospheric correction was completed to take into account site-specific terrain variability in the 334 km2 survey area centered around Crested Butte, CO. The atmospheric correction was completed using ACORN atmospheric correction software executed on 200 x 200 pixel kernels rather than line by line in order to capture local flight altitude conditions. Manual cloud delineation removed any small shaded areas that occurred within the data collection areas. Mosaics were developed based on preference for days that were in close timing proximity to the coincident ground campaign and with mosaicing criteria that minimized the angle between the sun and the sensor to retrieve the most consistent reflectance (min_phase_refl_tiled). Finally, we generated shade masks based on the geometry between the sun angle at time of flight, the ground surface, and the sensor. Here we provide the orthomosaiced estimated reflectance data (min_phase_refl_tiled.tif), canopy water content (wtrl), estimated atmospheric water vapor (wtrv), the observational data (obs) for this particular mosaic, the estimated visibility (vis), shade masks derived from the digital surface elevation model (shade) and the digital terrain model (shade_tch) added to the canopy height model (DOI: 10.15485/1617203), and a wavelength metadata file. Each image file is provided as a GeoTiff with internal tiling and LZW compression. These data can also be found on Google Earth Engine for extraction, download, and analysis of smaller extents (Google Earth Engine data is linked in external link table).

2018 NEON and 2025 CHESS Campaigns↗
DOE OSTI2023

Spatiotemporally Adaptive Compression for Scientific Dataset with Feature Preservation – A Case Study on Simulation Data with Extreme Climate Events Analysis

Scientific discoveries are increasingly constrained by limited storage space and I/O capacities. For time-series simulations and experiments, their data often need to be decimated over timesteps to accommodate storage and I/O limitations. In this paper, we propose a technique that addresses storage costs while improving post-analysis accuracy through spatiotemporal adaptive, error-controlled lossy compression. We investigate the trade-off between data precision and temporal output rates, revealing that reducing data precision and increasing timestep frequency lead to more accurate analysis outcomes. Additionally, we integrate spatiotemporal feature detection with data compression and demonstrate that performing adaptive error-bounded compression in higher dimensional space enables greater compression ratios, leveraging the error propagation theory of a transformation-based compressor. To evaluate our approach, we conduct experiments using the well-known E3SM climate simulation code and apply our method to compress variables used for cyclone tracking. Our results show a significant reduction in storage size while enhancing the quality of cyclone tracking analysis, both quantitatively and qualitatively, in comparison to the prevalent timestep decimation approach. Compared to three state-of-the-art lossy compressors lacking feature preservation capabilities, our adaptive compression framework improves perfectly matched cases in TC tracking by 26.4-51.3% at medium compression ratios and by 77.3-571.1% at large compression ratios, with a merely 5–11% computational overhead.

Gong, Qian↗
DOE OSTI2023

Direct prediction of inelastic neutron scattering spectra from the crystal structure*

Abstract Inelastic neutron scattering (INS) is a powerful technique to study vibrational dynamics of materials with several unique advantages. However, analysis and interpretation of INS spectra often require advanced modeling that needs specialized computing resources and relevant expertise. This difficulty is compounded by the limited experimental resources available to perform INS measurements. In this work, we develop a machine-learning based predictive framework which is capable of directly predicting both one-dimensional INS spectra and two-dimensional INS spectra with additional momentum resolution. By integrating symmetry-aware neural networks with autoencoders, and using a large scale synthetic INS database, high-dimensional spectral data are compressed into a latent-space representation, and a high-quality spectra prediction is achieved by using only atomic coordinates as input. Our work offers an efficient approach to predict complex multi-dimensional neutron spectra directly from simple input; it allows for improved efficiency in using the limited INS measurement resources, and sheds light on building structure-property relationships in a variety of on-the-fly experimental data analysis scenarios.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

Cascade of transitions in twisted and non-twisted graphene layers within the van Hove scenario

Abstract Motivated by measurements of compressibility and STM spectra in twisted bilayer graphene, we analyze the pattern of symmetry breaking for itinerant fermions near a van Hove singularity. Making use of an approximate SU(4) symmetry of the Landau functional, we show that the structure of the spin/isospin order parameter changes with increasing filling via a cascade of transitions. We compute the feedback from different spin/isospin orders on fermions and argue that each order splits the initially 4-fold degenerate van Hove peak in a particular fashion, consistent with the STM data and compressibility measurements, providing a unified interpretation of the cascade of transitions in twisted bilayer graphene. Our results follow from a generic analysis of an SU(4)-symmetric Landau functional and are valid beyond a specific underlying fermionic model. We argue that an analogous van Hove scenario explains the cascade of phase transitions in non-twisted Bernal bilayer and rhombohedral trilayer graphene.

75 CONDENSED MATTER PHYSICS, SUPERCONDUCTIVITY AND↗
DOE OSTI2019

Mesoscale Numerical Characterization of Kevlar and Carbon–Kevlar Hybrid Plain-Woven Fabric Compression Behavior

A mesoscale finite element model (FEM) of the plain-woven fabric representative elementary volume (REV/unit cell) is developed. The developed FEM is used to predict the nonlinear compaction behavior of monolithic and intraply hybrid fabric made of high-strength carbon and Kevlar fibers. For geometrical modeling of the fabric REV, TexGen, an open-source geometrical modeling software (developed at Nottingham University), is utilized, and for mechanical characterization, FE simulation tool Abaqus{sup ®} is used. Geometrical nonlinearity of the fabric, including periodic boundary conditions and yarn interactions, is incorporated in the REV FE model. Material nonlinearity caused by transversely isotropic behavior of fibrous yarns is implemented through the user material subroutine form in FE simulation along with local orientation assigned to each yarn in the REV. Material properties of fibers are extracted from the literature and supplier data. The compression behavior of monolithic Kevlar and hybrid carbon–Kevlar (carbon warp and Kevlar weft yarns, C–K) fabric is predicted using the proposed FE model. FEM-predicted results were found to have good agreement with the experimentally obtained ones. Results depict that C–K hybrid fabrics have improved compression strength. Hybridization effect is governed by yarns mutual interaction, yarn area at crossovers, and transverse in-plane properties of the interyarn hybridized fabric. The proposed FE model can be used to predict the mechanical behavior of woven fabrics with different weaving patterns, unit cell geometry, material properties, and loading conditions.

36 MATERIALS SCIENCE↗
DOE OSTI2017

Understanding the Hierarchy of Dense Subgraphs in Stationary and Temporally Varying Setting

Graphs are widely used to model relationships in a wide variety of domains such as sociology, bioinformatics, infrastructure, the WWW, to name a few. One of the key observations is that while real-world graphs are often globally sparse, they are locally dense. In other words, the average degree is often quite small (say at most 10 in a million vertex graph), but vertex neighborhoods are often dense. Finding dense subgraphs is a critical aspect of graph mining It has been used for finding communities and spam link farms in web graphs, graph visualization, real-time story identification, DNA motif detection in biological networks, finding correlated genes, epilepsy prediction, finding price value motifs in financial data, graph compression, distance query indexing, and increasing the throughput of social networking site servers. However, most standard formulations of this problem (like clique, quasi-clique, k-densest subgraph) are NP-hard. Furthermore, current dense subgraph finding algorithms usually optimize some objective, and only find a few such subgraphs without providing any structural relations, whereas the goal is rarely to find the "true optimum," but to identify many (if not all) dense substructures, understand their distribution in the graph, and ideally determine relationships among them. In this project, we first aim to devise algorithms and provide 3 implementations with nice visualizations to find the hierarchy between dense subgraphs, and then understand the structure of the hierarchy to gain more insight on the hidden patterns in real-world networks. Another important aspects in graph analysis is the temporal nature of networks. Networks evolve over time and in many applications data arrives at a high velocity, and thus it is important to design algorithms that can process data efficiently. We report three main results towards identifying dense structures in large evolving graphs. First, we will show how the hierarchical connectedness structure can be maintained efficiently, where connectedness is defined by increasing levels of connectivity strength. Next, we present dense structure can be identified in bipartite graphs without building projection graphs. And finally, we present a new method for peeling algorithms This new approach avoids sequential nature of peeling algorithms and is amenable to parallelization, which is crucial for processing high velocity data.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

ROOT I/O compression improvements for HEP analysis

We overview recent changes in the ROOT I/O system, increasing performance and enhancing it and improving its interaction with other data analysis ecosystems. Both the newly introduced compression algorithms, the much faster bulk I/O data path, and a few additional techniques have the potential to significantly to improve experiment's software performance. The need for efficient lossless data compression has grown significantly as the amount of HEP data collected, transmitted, and stored has dramatically increased during the LHC era. While compression reduces storage space and, potentially, I/O bandwidth usage, it should not be applied blindly: there are significant trade-offs between the increased CPU cost for reading and writing files and the reduce storage space.

Shadura, Oksana↗
DOE OSTI2021

Poisson’s ratio of eTPU molded bead foams in compression via in situ synchrotron X-ray microtomography

In situ synchrotron X-ray microtomography was used to characterize the bulk deformation behavior by computing the Poisson's ratio of expanded thermoplastic polyurethane (eTPU) molded bead foams used in footwear midsole during compression. Quantitative data on morphological characteristics were obtained using an iterative image processing workflow. Image correlation on the 4D datasets using DVC was performed to calculate the volumetric and axial strain to estimate the Poisson ratio. Strain maps from DVC showed the influence of variability in ligament thickness distribution on the global mechanical behavior exhibited which dominated the response seen in these bead foams. Lastly, our results showed a strong correlation between Poisson ratio and distribution of ligament thickness in foams.

36 MATERIALS SCIENCE↗
DOE OSTI2015

Parallel Tensor Compression for Large-Scale Scientific Data.

As parallel computing trends towards the exascale, scientific data produced by high-fidelity simulations are growing increasingly massive. For instance, a simulation on a three-dimensional spatial grid with 512 points per dimension that tracks 64 variables per grid point for 128 time steps yields 8 TB of data. By viewing the data as a dense five way tensor, we can compute a Tucker decomposition to find inherent low-dimensional multilinear structure, achieving compression ratios of up to 10000 on real-world data sets with negligible loss in accuracy. So that we can operate on such massive data, we present the first-ever distributed memory parallel implementation for the Tucker decomposition, whose key computations correspond to parallel linear algebra operations, albeit with nonstandard data layouts. Our approach specifies a data distribution for tensors that avoids any tensor data redistribution, either locally or in parallel. We provide accompanying analysis of the computation and communication costs of the algorithms. To demonstrate the compression and accuracy of the method, we apply our approach to real-world data sets from combustion science simulations. We also provide detailed performance results, including parallel performance in both weak and strong scaling experiments.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2016

Scalable subsurface inverse modeling of huge data sets with an application to tracer concentration breakthrough data from magnetic resonance imaging

When characterizing subsurface properties is crucial for reliable and cost-effective groundwater supply management and contaminant remediation. With recent advances in sensor technology, large volumes of hydro-geophysical and geochemical data can be obtained to achieve high-resolution images of subsurface properties. However, characterization with such a large amount of information requires prohibitive computational costs associated with “big data” processing and numerous large-scale numerical simulations. To tackle such difficulties, the Principal Component Geostatistical Approach (PCGA) has been proposed as a “Jacobian-free” inversion method that requires much smaller forward simulation runs for each iteration than the number of unknown parameters and measurements needed in the traditional inversion methods. PCGA can be conveniently linked to any multi-physics simulation software with independent parallel executions. In our paper, we extend PCGA to handle a large number of measurements (e.g. 106 or more) by constructing a fast preconditioner whose computational cost scales linearly with the data size. For illustration, we characterize the heterogeneous hydraulic conductivity (K) distribution in a laboratory-scale 3-D sand box using about 6 million transient tracer concentration measurements obtained using magnetic resonance imaging. Since each individual observation has little information on the K distribution, the data was compressed by the zero-th temporal moment of breakthrough curves, which is equivalent to the mean travel time under the experimental setting. Moreover, only about 2,000 forward simulations in total were required to obtain the best estimate with corresponding estimation uncertainty, and the estimated K field captured key patterns of the original packing design, showing the efficiency and effectiveness of the proposed method. This article is protected by copyright. All rights reserved.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2018

A comparative study of multi-material data structures for computational physics applications

The data structures used to represent the multi-material state of a computational physics application can have a significant impact on its performance. Here, we look at efficient data structures for applications where there may be many materials overall, but where most computational cells contain only one or a few of these materials. We develop simple performance models for selecting possible data structures and programming patterns. We verify the analytic performance models with a small test program of the representative cases. We discuss the impact of these techniques and analysis for multi-material physics applications, which are also applicable to a wide range of sparse computational data structures.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

The Complicated Link Between Material Properties and Microfracture Density for an Underground Explosion in Granite

Geophysical techniques are often implemented as quick and inexpensive ways to locate and characterize fractures in the subsurface, which is important for a number of geoscience fields. Seismic velocities are the most widely used proxies for identification of fractures, but the correlation is not always well-defined. In this study we present material property data: unconfined compressive strength (UCS), bulk density (ρ), Young's modulus (E), Poisson's ratio (ν), P wave velocity (V p ), and S wave velocity (V s ), in conjunction with microfracture densities measured on samples of granite collected before and after underground chemical explosions. Results indicate the relationship between fractures and material properties is complex, even in this single-lithology environment. We interpret that this complexity arises from varying fracture mechanisms (e.g. dilation-inducing fractures vs compression-inducing fractures) in different parts of the core, due to differences in stress conditions. Additional complexity may result from chemical interactions between the fresh fractures and the fluids in the area. Water content appears to have a significant, if not dominant, role in the unconfined compressive strength (UCS) of the samples. We suggest caution when using elastic property measurements as a proxy for fracturing in areas of explosion-induced damage, or in other areas where a variety of mechanisms induce fracturing.

58 GEOSCIENCES↗
DOE OSTI2016

Mathematical modelling of the beam under axial compression force applied at any point – the buckling problem

The study is devoted to stability of simply supported beam under axial compression. The beam is subjected to an axial load located at any point along the axis of the beam. The buckling problem has been desribed and solved mathematically. Critical loads have been calculated. In the particular case, the Euler’s buckling load is obtained. Explicit solutions are given. The values of critical loads are collected in tables and shown in figure. The relation between the point of the load application and the critical load is presented.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

High-speed x-ray imaging with the Keck pixel array detector (Keck PAD) for time-resolved experiments at synchrotron sources

Modern storage rings are readily capable of providing intense x-ray pulses, tens of picoseconds in duration, millions of times per second. Exploiting the temporal structure of these x-ray sources opens avenues for studying rapid structural changes in materials. Many processes (e.g. crack propagation, deformation on impact, turbulence, etc.) differ in detail from one sample trial to the next and would benefit from the ability to record successive x-ray images with single x-ray sensitivity while framing at 5 to 10 MHz rates. To this end, we have pursued the development of fast x-ray imaging detectors capable of collecting bursts of images that enable the isolation of single synchrotron bunches and/or bunch trains. The detector technology used is the hybrid pixel array detector (PAD) with a charge integrating front-end, and high-speed, in-pixel signal storage elements. A 384×256 pixel version, the Keck-PAD, with 150 µm × 150 µm pixels and 8 dedicated in-pixel storage elements is operational, has been tested at CHESS, and has collected data for compression wave studies. An updated version with 27 dedicated storage capacitors and identical pixel size has been fabricated.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2023

Cosmological constraints from the Subaru Hyper Suprime-Cam year 1 shear catalogue lensing convergence probability distribution function

Here we utilize the probability distribution function (PDF) of normalized convergence maps reconstructed from the Subaru Hyper Suprime-Cam (HSC) year 1 shear catalogue, in combination with the power spectrum, to measure the matter clustering amplitude S 8 = σ 8 $\sqrt {Ω_{m}/0.3}$. The large-scale structure’s statistical properties are incompletely described by the traditional two-point statistics, motivating our investigation of the PDF—a complementary higher-order statistic. By defining the PDF over the standard-deviation-normalized convergence map, we are able to isolate the non-Gaussian information. We use tailored simulations to compress the data vector and construct a likelihood approximation. We mitigate the impact of survey and astrophysical systematics with cuts on smoothing scales, redshift bins, and data vectors. We find S 8 = $0.860^{+0.066}_{–0.109}$ from the PDF alone and S 8 = 0.798$^{+0.029}_{–0.042}$ from the combination of the PDF and power spectrum (68% confidential level (CL)). The PDF improves the power-spectrum-only constraint by about 10%.

79 ASTRONOMY AND ASTROPHYSICS↗