Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “data reduction techniques”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2018

Data Reduction Techniques for Simulation, Visualization and Data Analysis

Data reduction is increasingly being applied to scientific data for numerical simulations, scientific visualizations, and data analyses. It is most often used to lower I/O and storage costs, and sometimes to lower in-memory data size as well. With this work, we consider five categories of data reduction techniques based on their information loss: 1) truly lossless, 2) near lossless, 3) lossy, 4) mesh reduction, and 5) derived representations. We then survey available techniques in each of these categories, summarize their properties from a practical point of view, and discuss relative merits within a category. We believe, in total, this work will enable simulation scientists and visualization/data analysis scientists to decide which data reduction techniques will be most helpful for their needs.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

HPDR: High-Performance Portable Scientific Data Reduction Framework

The rapid growth in scientific data generation is outpacing advancements in computing systems necessary for efficient storage, transfer, and analysis, particularly in the context of exascale computing. With the deployment of first-generation exascale computing systems and next-generation experimental facilities, this gap is widening and necessitates effective data reduction techniques to manage enormous data volumes. Over the past decade, various data reduction methods, including lossless compression, error-controlled lossy compression, and data refactoring, have been developed to accelerate I/O in scientific workflows. Despite significant reductions in data volume, these methods introduce considerable computational overhead, which can become the new bottleneck in data processing. To mitigate this, GPU-accelerated data reduction algorithms have been introduced. However, challenges remain in their integration into exascale workflows, including limited portability across different GPU architectures, substantial memory transfer overhead, and reduced scalability on dense multi-GPU systems. To address these challenges, we propose HPDR, a high-performance and portable data reduction framework. HPDR is designed to enable the execution of state-of-the-art reduction algorithms across diverse processor architectures while reducing memory transfer overhead to 2.3 % of the original, resulting in up to 3.5× faster throughput compared to existing solutions. It also achieves up to 96% of the theoretical speedup in multi-GPU settings. In addition, evaluations on accelerating I/O operations at scale up to 1,024 nodes of the Frontier supercomputer demonstrate that HPDR can achieve up to 103 TB/s reduction throughput, providing up to 4× acceleration in parallel I/O performance compared to existing data reduction routines. This work highlights the potential of HPDR to significantly enhance data reduction efficiency in exascale computing environments.

Chen, Jieyang [University of Oregon]↗
DOE OSTI2025

Machine Learning Techniques for Data Reduction of Climate Applications

Scientists conduct large-scale simulations to compute derived quantities-of-interest (QoI) from primary data. Often, QoI are linked to specific features, regions, or time intervals, such that data can be adaptively reduced without compromising the integrity of QoI. For many spatiotemporal applications, these QoI are binary in nature and represent presence or absence of a physical phenomenon. We present a pipelined compression approach that first uses neural-network-based techniques to derive regions where QoI are highly likely to be present. Then, we employ a Guaranteed Autoencoder (GAE) to compress data with differential error bounds. GAE uses QoI information to apply low-error compression to only these regions. This results in overall high compression ratios while still achieving downstream goals of simulation or data collections. Experimental results are presented for climate data generated from the E3SM Simulation model for downstream quantities such as tropical cyclone and atmospheric river detection and tracking. These results show that our approach is superior to comparable methods in the literature.

Li, Xiao [University of Florida]↗
DOE OSTI2022

Maintaining Trust in Reduction: Preserving the Accuracy of Quantities of Interest for Lossy Compression

As the growth of data sizes continues to outpace computational resources, there is a pressing need for data reduction techniques that can significantly reduce the amount of data and quantify the error incurred in compression. Compressing scientific data presents many challenges for reduction techniques since it is often on non-uniform or unstructured meshes, is from a high-dimensional space, and has many Quantities of Interests (QoIs) that need to be preserved. To illustrate these challenges, we focus on data from a large scale fusion code, XGC. XGC uses a Particle-In-Cell (PIC) technique which generates hundreds of PetaBytes (PBs) of data a day, from thousands of timesteps. XGC uses an unstructured mesh, and needs to compute many QoIs from the raw data, f.One critical aspect of the reduction is that we need to ensure that QoIs derived from the data (density, temperature, flux surface averaged momentums, etc.) maintain a relative high accuracy. We show that by compressing XGC data on the high-dimensional, nonuniform grid on which the data is defined, and adaptively quantizing the decomposed coefficients based on the characteristics of the QoIs, the compression ratios at various error tolerances obtained using a multilevel compressor (MGARD) increases more than ten times. We then present how to mathematically guarantee that the accuracy of the QoIs computed from the reduced f is preserved during the compression. We show that the error in the XGC density can be kept under a user-specified tolerance over 1000 timesteps of simulation using the mathematical QoI error control theory of MGARD, whereas traditional error control on the data to be reduced does not guarantee the accuracy of the QoIs.

Gong, Qian↗
DOE OSTI2023

Scalable Incremental Checkpointing using GPU-Accelerated De-Duplication

Writing large amounts of data concurrently to stable storage is a typical I/O pattern of many HPC workflows. This pattern introduces high I/O overheads and results in increased storage space utilization especially for workflows that need to capture the evolution of data structures with high frequency as checkpoints. In this context, many applications, such as graph pattern matching, perform sparse updates to large data structures between checkpoints. For these applications, incremental checkpointing techniques that save only the differences from one checkpoint to another can dramatically reduce the checkpoint sizes, I/O bottlenecks, and storage space utilization. However, such techniques are not without challenges: it is non-trivial to transparently determine what data has changed since a previous checkpoint and assemble the differences in a compact fashion that does not result in excessive metadata. State-of-art data reduction techniques (e.g., compression and de-duplication) have significant limitations when applied to modern HPC applications that leverage GPUs: slow at detecting the differences, generate a large amount of metadata to keep track of the differences, and ignore crucial spatiotemporal checkpoint data redundancy. This paper addresses these challenges by proposing a Merkle tree-based incremental checkpointing method to exploit GPUs' high memory bandwidth and massive parallelism. Experimental results at scale show a significant reduction of the I/O overhead and space utilization of checkpointing compared with state-of-the-art incremental checkpointing and compression techniques.

Tan, Nigel↗
DOE OSTI2019

Machine-Learning Analysis of Moisture Carryover in Boiling Water Reactors

Moisture carryover (MCO) is modeled in the General Electric Type-4 boiling water reactor (BWR) using machine-learning methods and data from operating plants. Understanding MCO and the conditions that give rise to an elevated value is important since excessive MCO can damage critical turbine components, can result in elevated dose levels to on-site personnel, and can interfere with late-cycle power management. The analysis of MCO takes into account simplifying reactor symmetries and important geometric dependencies. The plant data are taken from several reactors and were collected over multiple years and multiple fuel cycles. A brief description of the origin of MCO in U.S. BWR plants is given. A machine-learning model is constructed from the data using applicable algorithms and data-reduction techniques. Matching model complexity with available data is one of the more challenging machine-learning tasks. Too many features and too little data will lead to overfitting. The data for each fuel cycle included over 6876 original features, 9 for each fuel bundle. Two approaches are used to reduce the data set into a manageable number of features. The first was an engineering analysis that resulted in the selection of steam quality Q and steam liquid phase velocity V L as the main features driving MCO. Using a Q and a V L for each fuel bundle gives 1528 Q and a V L feature describing the reactor behavior. An analysis of different functional forms of these two variables led to the actual inputs to the neural network model. The second approach involved the use of statistical techniques such as Pearson’s correlation and k-means analysis. The identified groupings of bundles behaved similarly. Treating each grouping as a single feature further reduced the input variable set to a manageable number. Here, a model selection criterion is proposed, and results are presented along with a discussion of related issues.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

Data reduction for low energy nuclear physics experiments using data frames

Low energy nuclear physics experiments are transitioning towards fully digital data acquisition systems. Realizing the gains in flexibility afforded by these systems relies on equally flexible data reduction techniques. In this paper, methods utilizing data frames and in-memory techniques to work with data, including data from self-triggering, digital data acquisition systems, are discussed within the context of a Python package, sauce. It is shown that data frame operations can encompass common analysis needs and allow interactive data analysis. Two event building techniques, dubbed referenced and referenceless event building, are shown to provide a means to transform raw list mode data into correlated multi-detector events. These techniques are demonstrated in the analysis of two example data sets.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2024

Codebase release 2.0 for sauce

Low energy nuclear physics experiments are transitioning towards fully digital data acquisition systems. Realizing the gains in flexibility afforded by these systems relies on equally flexible data reduction techniques. In this paper, methods utilizing data frames and in-memory techniques to work with data, including data from self-triggering, digital data acquisition systems, are discussed within the context of a Python package, sauce. It is shown that data frame operations can encompass common analysis needs and allow interactive data analysis. Two event building techniques, dubbed referenced and referenceless event building, are shown to provide a means to transform raw list mode data into correlated multi-detector events. These techniques are demonstrated in the analysis of two example data sets.

Marshall, Caleb (ORCID:0000000211942920)↗
DOE OSTI2021

Prospects for Galactic and stellar astrophysics with asteroseismology of giant stars in the TESS continuous viewing zones and beyond

ABSTRACT The NASA Transiting Exoplanet Survey Satellite (NASA-TESS) mission presents a treasure trove for understanding the stars it observes and the Milky Way, in which they reside. We present a first look at the prospects for Galactic and stellar astrophysics by performing initial asteroseismic analyses of bright (G < 11) red giant stars in the TESS southern continuous viewing zone (SCVZ). Using three independent pipelines, we detect νmax and Δν in 41 per cent of the 15 405 star parent sample (6388 stars), with consistency at a level of $\sim \! 2{{\ \rm per\ cent}}$ in νmax and $\sim \! 5{{\ \rm per\ cent}}$ in Δν. Based on this, we predict that seismology will be attainable for ∼3 × 105 giants across the whole sky and at least 104 giants with ≥1 yr of observations in the TESS-CVZs, subject to improvements in analysis and data reduction techniques. The best quality TESS-CVZ data, for 5574 stars where pipelines returned consistent results, provide high-quality power spectra across a number of stellar evolutionary states. This makes possible studies of, for example, the asymptotic giant branch bump. Furthermore, we demonstrate that mixed ℓ = 1 modes and rotational splitting are cleanly observed in the 1-yr data set. By combining TESS-CVZ data with TESS-HERMES, SkyMapper, APOGEE, and Gaia, we demonstrate its strong potential for Galactic archaeology studies, providing good age precision and accuracy that reproduces well the age of high [α/Fe] stars and relationships between mass and kinematics from previous studies based on e.g. Kepler. Better quality astrometry and simpler target selection than the Kepler sample makes this data ideal for studies of the local star formation history and evolution of the Galactic disc. These results provide a strong case for detailed spectroscopic follow-up in the CVZs to complement that which has been (or will be) collected by current surveys.

Mackereth, J. Ted↗
DOE OSTI2022

Information-Theoretic Exploration of Multivariate Time-Varying Image Databases

Modern scientific simulations produce very large datasets, making interactive exploration of such data computationally prohibitive. An increasingly common data reduction technique is to store visualizations and other data extracts in a database. The Cinema project is one such approach, storing visualizations in an image database for post hoc exploration and interactive image-based analysis. This work focuses on developing efficient algorithms that can quantify various types of multivariate dependencies existing within multi-variable datasets. It applies specific mutual information measures for the quantification of salient regions from multivariate image data. Here, using such information measures, the opacity of the images is modulated so that the salient regions are automatically highlighted and the domain scientists can interactively explore the most relevant regions for scientific discovery.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

PROTEUS: Machine Learning Driven Resilience for Extreme-scale Systems

The objective of this project is to design, develop, and evaluate scalable software to enhance resilience, data checkpointing, program restart, and analysis. The proposed tasks are to 1) develop scalable machine learning techniques to learn temporal change patterns in a scalable and in-situ manner, and to minimize data movement and maximize learning locally closest to data; 2) design a concise data representation and indexing mechanism to capture the distribution of changes in data that can guarantee point-wise user-defined tolerable errors while reducing the data storage requirements by an order of magnitude or more; 3) develop data reduction techniques as library modules; 4) exploit local SSD for minimizing data movement in storage hierarchy; 5) develop anomaly detection algorithms that can predict corruptions based on learning of emerging patterns; 6) develop software libraries to be incorporated within widely used data formats and APIs; and 7) evaluate the proposed software using DOE scientific applications. The outcomes of the proposed work are to satisfy many synergistic data reduction and resilience requirements for large-scale data intensive applications executed on extreme-scale computing systems. The developed mechanism for error-bound data approximation is directly applicable to existing scientific applications. Through machine learning from historical events and change distribution, this work will enable anomaly detection for DOE computer facility.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

A Digital Twin Framework Utilizing Machine Learning for Robust Predictive Maintenance: Enhancing Tire Health Monitoring

We introduce a novel digital twin (DT) framework for the predictive maintenance of long-term physical systems. Using monitoring tire health as an application, we show how the DT framework can be used to enhance automotive safety and efficiency, and how the technical challenges can be overcome using a three-step approach. First, to manage the data complexity over a long operation span, we employ data reduction techniques to concisely represent physical tires using historical performance and usage data. Relying on these data, for fast real-time prediction, we train a transformer-based model offline on our concise dataset to predict future tire health over time, represented as remaining casing potential (RCP). Based on our architecture, our model quantifies both epistemic and aleatoric uncertainties, providing reliable confidence intervals around predicted RCP. Second, to incorporate real-time data, we update the predictive model in the DT framework, ensuring its accuracy throughout its lifespan with the aid of hybrid modeling and the use of the discrepancy function. Third, to assist decision-making in predictive maintenance, we implement a tire state decision algorithm, which strategically determines the optimal timing for tire replacement based on RCP forecasted by our transformer model. This approach ensures that our DT accurately predicts system health, continually refines its digital representation, and supports predictive maintenance decisions. Furthermore, our framework effectively embodies a physical system, leveraging big data and machine learning (ML) for predictive maintenance, model updates, and decision-making.

advanced computing infrastructure↗
DOE OSTI2025

Bringing Different Views Together: A Hybrid Cooperative Perception Framework for Connected Autonomous Vehicles

Cooperative perception will be essential for connected autonomous vehicles to enhance object recognition and optimize path planning by sending data information about the surrounding environment. However, an inherent challenge in existing systems is the high bandwidth cost of transmitting information in real-time, which restricts cooperative perception’s practicality. Here, this work presents a hybrid cooperative perception fusion framework aimed at mitigating this issue by optimizing data transmission according to available bandwidth or through data reduction techniques. Our methods ensure that vehicles can rapidly transmit high-confidence data without overwhelming the network. Experimental results indicate that our methodology substantially diminishes data transmission sizes while maintaining object detection accuracy. For cooperative perception in autonomous vehicle systems, our approach provides a scalable and effective way to get past the bandwidth barrier.

Carrillo, Dominic [Univ. of North Texas, Denton, T↗
DOE OSTI2023

Data reduction considerations for the burning velocity of spherical constant volume flames of R32 (CH 2 F 2 ) with air

Here, the present work explores data reduction techniques for the measurement of the laminar burning velocities of R32(CH 2 F 2 )-air mixtures using a constant volume combustion device, in which the pressure-time history is the only measured parameter. To allow clear assessment of the accuracy of the data reduction methods, the pressure-time histories used for analysis are synthetically generated via a detailed numerical simulation employing full kinetics and with and without an optically-thin radiation model. Various data reduction models are employed, including a two-zone model and two multi-zone models, and these are compared with the results from the burning velocity obtained from the output of the numerical simulation. The data reduction schemes are shown to be accurate if the same radiation model is employed in the data reduction as was used in the flame simulation to generate the pressure trace used for post-processing. If the incorrect radiation model is employed, however, the errors can be quite large. The effects of stretch, radiation, and different data post-processing methodologies are explored and the errors quantified. Stretch is shown to be important for the early stages and the selected data range that is used for extrapolation has a significant effect on the extrapolated burning velocity. However, with an appropriate choice of data considered for extrapolation, the prediction of the unstretched burning velocity can be quite accurate.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Magnesium isotopes of the bulk solar wind from Genesis diamond-like carbon films

–NASA’s Genesis Mission returned solar wind (SW) to the Earth for analysis to derive the composition of the solar photosphere from solar material. SW analyses control the precision of the derived solar compositions, but their ultimate accuracy is limited by the theoretical or empirical models of fractionation due to SW formation. Mg isotopes are “ground truth” for these models since, except for CAIs, planetary materials have a uniform Mg isotopic composition (within ≤1&) so any significant isotopic fractionation of SW Mg is primarily that of SW formation and subsequent acceleration through the corona. This study analyzed Mg isotopes in a bulk SW diamond-like carbon (DLC) film on silicon collector returned by the Genesis Mission. A novel data reduction technique was required to account for variable ion yield and instrumental mass fractionation (IMF) in the DLC. The resulting SW Mg fractionation relative to the DSM-3 laboratory standard was ( 14.4&, 30.2&) (4.1&, 5.5&), where the uncertainty is 2σ SE of the data combined with a 2.5& (total) error in the IMF determination. Two of the SW fractionation models considered generally agreed with our data. Their possible ramifications are discussed for O isotopes based on the CAI nebular composition of McKeegan et al. (2011).

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2022

The public health exposome and pregnancy-related mortality in the United States: a high-dimensional computational analysis

Racial inequities in maternal mortality in the U.S. continue to be stark. The 2015–2018, 4-year total population, county-level, pregnancy-related mortality ratio (PRM; deaths per 100,000 live births; National Center for Health Statistics (NCHS), restricted use mortality file) was linked with the Public Health Exposome (PHE). Using data reduction techniques, 1591 variables were extracted from over 62,000 variables for use in this analysis, providing information on the relationships between PRM and the social, health and health care, natural, and built environments. Graph theoretical algorithms and Bayesian analysis were applied to PHE/PRM linked data to identify latent networks. PHE variables most strongly correlated with total population PRM were years of potential life lost and overall life expectancy. Population-level indicators of PRM were overall poverty, smoking, lack of exercise, heat, and lack of adequate access to food. In this high-dimensional analysis, overall life expectancy, poverty indicators, and health behaviors were found to be the strongest predictors of pregnancy-related mortality. This provides strong evidence that maternal death is part of a broader constellation of both similar and unique health behaviors, social determinants and environmental exposures as other causes of death.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2018

Efficient Data-Driven Geologic Feature Characterization from Pre-stack Seismic Measurements using Randomized Machine-Learning Algorithm

Conventional seismic techniques for detecting the subsurface geologic features are challenged by limited data coverage, computational inefficiency and subjective human factors. We developed a novel data-driven geological feature characterization approach based on pre-stack seismic measurements. Our characterization method employs an efficient and accurate machine learning method to extract useful subsurface geologic features automatically. Specifically, we use kernel ridge regression to account for the nonlinear relationship between seismic data and geological features. We moreover employ kernel tricks to avoid the explicit nonlinear mapping and infinite dimension of feature space. Yet, the conventional kernel ridge regression can be computationally prohibitive because of the large volume of seismic measurements. We employ a data reduction technique in combination with the conventional kernel ridge regression method to improve the computational efficiency and reduce memory usage. In particular, we utilize a randomized numerical linear algebra technique, named Nyström method, to effectively reduce the dimensionality of the feature space without compromising the information content required for accurate characterization. We provide thorough computational cost analysis to show the efficiency of our new geological feature characterization methods. We validate the performance of our method in characterizing geologic fault zones because faults play an important role in various subsurface applications. Our numerical examples demonstrate that our new characterization method significantly improves the computational efficiency while maintaining comparable accuracy. Interestingly, we demonstrate that our approach yields a speedup ratio on the order of ~102 to ~103 in a multicore computational environment.

58 GEOSCIENCES↗