Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “evaluation datasets”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 649 records · Page 36
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

CGSim: A Simulation Framework for Large Scale Distributed Computing Environment

Large-scale distributed computing infrastructures such as the Worldwide LHC Computing Grid (WLCG) require comprehensive simulation tools for evaluating performance, testing new algorithms, and optimizing resource allocation strategies. However, existing simulators suffer from limited scalability, hardwired algorithms, lack of real-time monitoring, and inability to generate datasets suitable for modern machine learning approaches. We present CGSim, a simulation framework for large-scale distributed computing environments that addresses these limitations. Built upon the validated SimGrid simulation framework, CGSim provides high-level abstractions for modeling heterogeneous grid environments while maintaining accuracy and scalability. Key features include a modular plugin mechanism for testing custom workflow scheduling and data movement policies, interactive real-time visualization dashboards, and automatic generation of event-level datasets suitable for AI-assisted performance modeling. We demonstrate CGSim’s capabilities through a comprehensive evaluation using production ATLAS PanDA workloads, showing significant calibration accuracy improvements across WLCG computing sites. Scalability experiments show near-linear scaling for multi-site simulations, with distributed workloads achieving 6 × better performance compared to single-site execution. The framework enables researchers to simulate WLCG-scale infrastructures with hundreds of sites and thousands of concurrent jobs within practical time budget constraints on commodity hardware.

Vatsavai, Sairam Sri [Brookhaven National Laborato↗
DOE OSTI2021

Quantitative resilience evaluation on recovery from emergency situations in nuclear power plants

Here, this paper mainly introduces how to develop a resilience evaluation model by quantifying the relationship between resilience and resilience components in regard to recovering from emergency situations in nuclear power plants (NPPs). It is an extension of the author’s previous researches. To develop the model, the first step was to analyze event reports published in Republic of Korea in terms of resilience factors from different safety perspectives. Second, statistical methods such as factor analysis, principal component analysis, and multivariate regression analysis were applied to the dataset to identify relations among resilience factors and develop a quantitative resilience evaluation model for recovery from emergency situations in NPPs. Third, the model was validated using the recent event report data released in Korea. Lastly, we present a discussion on how to apply the quantitative resilience evaluation model to results from stress tests performed in Korea.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗
NASA NTRS2017

Metadata Evaluation and Improvement: Evolving Analysis and Reporting

ESIP Community members create and manage a large collection of environmental datasets that span multiple decades, the entire globe, and many parts of the solar system. Metadata are critical for discovering, accessing, using and understanding these data effectively and ESIP community members have successfully created large collections of metadata describing these data. As part of the White House Big Earth Data Initiative (BEDI), ESDIS has developed a suite of tools for evaluating these metadata in native dialects with respect to recommendations from many organizations. We will describe those tools and demonstrate evolving techniques for sharing results with data providers.

metadata recommendations↗
DOE OSTI2024

Shorter function summaries for finite state machine-based high consequence systems using logic synthesis and tautologies (Final Report LDRD 24-1302)

Computer programs are often viewed as collections of functions – each function has parameters (inputs) and computes a return value, and each has potential side effects that modify program state (outputs). In this research, a Sandia symbolic execution tool designed to support “human-in-the-loop” analysis was modified to automatically create “function summaries,” and a new tool, “diaboolical,” was created to support enhancing readability of the summary using a novel approach to bit-vector simplification that leverages logic synthesis and tautologies. For this effort, students at Auburn University created several finite state machines (FSMs) to serve as exemplars for high-consequence systems. Function summaries for each of the machines were obtained, and then portions of the summaries were simplified using both diaboolical and the simplification procedure of a popular SMT solver. A comparison of the results shows that diaboolical can often produce smaller function summaries, with expression length improvements over the unsimplified function summaries ranging from 0% to 90% for diaboolical and 0% to 65% for the SMT solver, though diaboolical had a significantly greater cost in time. Diaboolical was evaluated against a collection of “arbitrary” C-code as well as FSM exemplars, and for both datasets it achieved an approximately 10% improvement in expression length compared to simplifications that could be obtained using existing techniques. Function summaries can assist assurance efforts that evaluate existing systems and their executable code. A smaller function summary is likely easier for humans to understand and could thus increase the ability and efficacy of assurance practices centered around the analysis of executable artifacts.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

An overview of the Western United States Dynamically Downscaled Dataset (WUS-D3)

Abstract. Predicting future climate change over a region of complex terrain, such as the western United States (US), remains challenging due to the low resolution of global climate models (GCMs). Yet the climate extremes of recent years in this region, such as floods, wildfires, and drought, are likely to intensify further as climate warms, underscoring the need for high-quality and high-resolution predictions. Here, we present an ensemble of dynamically downscaled simulations over the western US from 1980–2100 at 9 km grid spacing, driven by 16 latest-generation GCMs. This dataset is titled the Western US Dynamically Downscaled Dataset (WUS-D3). We describe the challenges of producing WUS-D3, including GCM selection and technical issues, and we evaluate the simulations' realism by comparing historical results to temperature and precipitation observations. The future downscaled climate change signals are shaped in physically credible ways by the regional model's more realistic coastlines and topography. (1) The mean warming signals are heavily influenced by more realistic snowpack. (2) Mean precipitation changes are often consistent with wetting on the windward side of mountain complexes, as warmer, moister air masses are uplifted orographically during precipitation events. (3) There are large fractional precipitation increases on the lee side of mountain complexes, leading to potentially significant changes in water resources and ecology in these arid landscapes. (4) Increases in precipitation extremes are generally larger than in the GCMs, driven by locally intensified atmospheric updrafts tied to sharper, more realistic gradients in topography. (5) Changes in temperature extremes are different from what is expected by a shift in mean temperature and are shaped by local atmospheric dynamics and land surface feedbacks. Because of its high resolution, comprehensiveness, and representation of relevant physical processes, this dataset presents a unique opportunity to evaluate societally relevant future changes in western US climate.

Rahimi, Stefan (ORCID:0000000331884462)↗
DOE OSTI2025

HexWeather: Hexagonal Spatial Data Aggregation for Weather-Driven Grid Resilience Analysis

Extreme weather accounts for over 8 0 % of major U.S. power outages since 2000, highlighting the need for spatial tools that align weather data with the irregular boundaries of electric infrastructure. This paper introduces HexWeather, a modular, resolution-aware framework for aggregating historical and forecasted weather data using Uber's H3 hexagonal spatial indexing system. Unlike traditional methods that rely on state or county-level grids, HexWeather enables weather analysis across custom geographies such as utility service areas where public datasets are often unavailable or misaligned. Using Open-Meteo data, we evaluate how H3 resolution affects anomaly detection, spatial variability, and forecast uncertainty across three scales: state, county, and utility. Results show that while coarse resolutions suffice for broad trend tracking, finer resolutions are essential for identifying localized variability and operational risks. By applying metrics like Z-score standard deviation and interquartile range, HexWeather quantifies the spatial spread of both historical anomalies and forecasted conditions, allowing users to assess resolution adequacy for each analysis. This framework supports rapid weather data reuse, reproducible anomaly detection, and predictive modeling for infrastructure resilience. By bridging spatial misalignment in traditional datasets and enabling retrospective and forward-looking analysis within the same pipeline, HexWeather lays the groundwork for better post event analysis, outage prediction, and resilience planning.

Morris, Jacob [ORNL]↗
DOE OSTI2023

Accelerating Random Forest Classification on GPU and FPGA

Random Forests (RFs) are a commonly used machine learning method for classification and regression tasks spanning a variety of application domains, including bioinformatics, business analytics, and software optimization. While prior work has focused primarily on improving performance of the training of RFs, many applications, such as malware identification, cancer prediction, and banking fraud detection, require fast RF classification. In this work, we accelerate RF classification on GPU and FPGA. In order to provide efficient support for large datasets, we propose a hierarchical memory layout suitable to the GPU/FPGA memory hierarchy. We design three RF classification code variants based on that layout, and we investigate GPU- and FPGA-specific considerations for these kernels. Our experimental evaluation, performed on an Nvidia Xp GPU and on a Xilinx Alveo U250 FPGA accelerator card using publicly available datasets on the scale of millions of samples and tens of features, covers various aspects. First, we evaluate the performance benefits of our hierarchical data structure over the standard compressed sparse row (CSR) format. Second, we compare our GPU implementation with cuML, a machine learning library targeting Nvidia GPUs. Third, we explore the performance/accuracy tradeoff resulting from the use of different tree depths in the RF. Finally, we perform a comparative performance analysis of our GPU and FPGA implementations. Our evaluation shows that for high accuracy targets, our GPU implementation yields 5-9x speedup over CSR, and up to a 2x speedup over cuML.

FPGA, Xilinx FPGA, GPU, Random Forest classificati↗
NASA NTRSDate not supplied

The SCIFLI Airborne Observation of Artemis 1 Ascent

SCIFLI, The Scientifically Calibrated In-Flight Imagery (https://scifli.larc.nasa.gov/), team at NASA Langley Research Center specializes in the collection of multispectral data for space vehicles during Ascent and Entry, Descent, and Landing (EDL) observations. Multispectral datasets are invaluable across the NASA Agency and to commercial stakeholders for evaluating the performance of launch and re-entry space vehicles and ensuring the safety of the scientific research community. The launch of Artemis 1 marked the initiation of NASA returning to lunar exploration. SCIFLI’s Airborne Multispectral Imager (SAMI) was deployed by the SCIFLI team to scientifically document the historic event and provide the NASA research community with aerial footage of the event in wavelength channels ranging from the ultraviolet (UV) to the visible (VIS) to mid-wave infrared (MWIR). SAMI was designed to spectrally image unique aerothermal phenomena during the Artemis 1 launch. The use of these datasets captured in-flight during the launch will provide insight to research organizations across the Agency; aiding in validation efforts for simulations and modeling that contributed to the kickoff of the Agency’s resurgence to lunar exploration. This presentation will focus on the imagery captured on SAMI by the WB-57 team on November 16th, 2022, during the observation. Imaging objectives were identified and considered beforehand to determine the configuration of the SAMI instrument for the imaging mission. SCIFLI and Opto-Knowledge Systems, Inc. (OKSI) performed a thorough review of the datasets collected to identify and characterize aerothermal phenomena occurring during the observation. Additional post-processing was completed to provide quantitatively calibrated temperature images of the rocket during the observation. Some of the candidates were not optimal for quantitative temperature extraction due to common degradation factors, but after various image enhancements they proved useful for qualitatively characterizing different phenomena during the observation.

Artemis1↗
NASA NTRSDate not supplied

Evaluation of Spectral Band Adjustment Factors for Cross-Calibration of Visible Imagers

The CERES EBAF dataset provides TOA SW and LW fluxes for long-term monitoring of the Earth’s energy balance and to validate climate models. The EBAF products, based on the Terra and Aqua CERES instrument observed radiances, rely on coincident measurements from the onboard MODIS imagers to determine cloud properties that are used in the angular distribution model scene selection required to convert the CERES observed radiances into flux values. Once the Terra and Aqua orbits start drifting outside of their 15-minute window, the CERES project will rely solely on the NOAA-20 CERES observations. A seamless transition of fluxes and clouds can only occur if the analogous MODIS and VIIRS channels are properly inter-calibrated. The visible band (0.65 µm) in MODIS (band 1) and VIIRS (M5 or I1) is critical for retrieving the cloud mask and optical depth. The spectral response functions (SRF) of these bands differ noticeably and will require scene dependent spectral band adjustment factors (SBAF) for proper radiometric scaling between them. Fortunately, the VIIRS I1 and M5 bands are calibrated using the same solar diffuser as a reference, which implies that band reflectance differences must be due to the spectral disparity. The coincident I1 and M5 TOA reflectance measurements provide the optimal opportunity to validate SBAFs over many surface and cloud conditions. The CERES project maintains SCIAMACHY, GOME-2, Hyperion, and radiative transfer model-based scene-stratified hyper-spectral reflectance measurements that can be convolved with sensor pair SRFs to compute the corresponding SBAF. This study will evaluate the SCIAMACHY, GOME-2, Hyperion, and RTM based SBAFs over tropical ocean targets used in SNO inter-calibration of MODIS and VIIRS, including clear-sky ocean, deep convective clouds (DCC), and liquid cloud targets in terms of their applicability to absolute intercomparison of visible imagers. The large SCIAMACHY and GOME-2 footprints and the lack of Hyperion cloudy scenes will impact the resulting SBAFs and necessitates the need for future CLARREO hyper-spectral reflectances to decrease SBAF uncertainties Preliminary results indicate that the SCIAMACHY and GOME-2 based SBAFs for the VIIRS I1 and M5 band pairs may differ by 1.5% for some ATO scene types. Because most of the modern GEOs visible band SRFs encompass the VIIRS I1 band SRF, these evaluations are critical to ensure that the MODIS, VIIRS, and GEO clouds and fluxes are consistent. SBAFs for Libya-4, Dome-C and other Earth invariant target will also be evaluated.

D. R. Doelling↗
NASA NTRSDate not supplied

Batch Effect Correction Methods for NASA GeneLab Transcriptomic Datasets

RNA sequencing (RNA-seq) data from space biology experiments promise to yield invaluable insights into the effects of spaceflight on terrestrial biology. However, sample numbers from each study are low due to limited crew availability, hardware, and space. To increase statistical power, spaceflight RNA-seq datasets from different missions are often aggregated together. However, this can introduce technical variation or "batch effects", often due to differences in sample handling, sample processing, and sequencing platforms. Several computational methods have been developed to correct for technical batch effects, thereby reducing their impact on true biological signals. In this study, we combined 7 mouse liver RNA-seq datasets from NASA GeneLab (part of the NASA Open Science Data Repository) to evaluate several common batch effect correction methods (ComBat and ComBat-seq from the sva R package, and Median Polish, Empirical Bayes, and ANOVA from the MBatch R package). We quantitatively evaluated the ability of these methods to correct for technical batch variables in space biology RNA-seq data using the following criteria: BatchQC, principal component analysis, dispersion separability criterion, log fold change correlation, and differential gene expression analysis. Each batch variable / correction method combination was then assessed using a custom scoring approach to identify the optimal correction method for the combined dataset, by geometrically probing the space of all allowable scoring functions to yield an aggregate volume-based scoring measure. Finally, we describe the way in which the GeneLab multi-study analysis and visualization portal will allow users to examine the presence or absence of batch effects using multiple metrics. If the user chooses to perform batch effect correction, the scoring approach described here can be implemented to identify the optimal correction method to use for their specific combined dataset prior to analysis.

Lauren M. Sanders↗
NASA NTRS2024

The Radio Fundamental Catalogue. I. Astrometry

We present the all-sky catalogue of absolute positions and estimates of correlated flux density of 21,942 compact radio sources determined from processing interferometric visibility data of virtually all VLBI observing sessions at 2 to 23~GHz from 72 programs suitable for absolute astrometry collected for 30 years. We used a novel technique of generation of a dataset of fused observables that allowed us to incorporate all available data in our analysis. The catalogue is the most complete and most precise to date. It forms the foundation and reference for positional astronomy, space geodesy, space navigation, population analysis of active galactic nuclei (AGNs), and provides calibrators for phase-referencing for differential astrometry and VLBI astrophysical observations. Its accuracy was evaluated through a detailed accounting systematic errors, rigorous decimation tests, comparison of different datasets, and comparison with other catalogues. The catalogue samples preferentially the AGNs with a strong contemporary parsec-scale synchrotron emission. Its milliarcsecond level positional accuracy allows association of these AGNs with detections in a wide range of the electromagnetic spectrum from low-frequency radio to gamma-rays and high energy neutrinos. We describe the innovative data processing and calibration technique in full detail, report the in depth analysis of random and systematic positional errors, and provide the list of associations with large surveys at different wavelengths.

Astronomy↗
DOE OSTI2020

Multitask Recommender Systems for Cancer Drug Response

The problem we are currently trying to address is that there are many types of cancer drugs and many types of cancers and there is not always experimental data for a specific cancer type and cancer drug interaction. While there is a large possible set of feasible drug and cancer combinations, testing each pair is not realistic due to the high monetary cost of cell-based assays. Thus, this leaves researchers with a difficult choice of what drugs they should test on specific cancer types. This issue is known as the cold-start problem. Our focus is on developing recommender systems capable of addressing the cold-start problem as it relates to interaction between cancer types and cancer drugs. One of the most effective ways to address the cold-start problem is through large data analysis, however due to the cost prohibitive nature of cancer research the largest available data set size is the Genomics of Drug Sensitivity in Cancer with 494,973 genomic associations. To achieve optimal model performance on the cold-start problem, it is advantageous to employ multitask algorithms that are capable of transferring information between cancer datasets. The aim of this report is to draw from adaptations and state of the art developments in both algorithms for recommender systems and multitask learning to model the interaction between cancer cell lines and cancer drugs. Cancer cell lines are defined by the US National Cancer Institute as "cancer cells that keep dividing and growing over time, under certain conditions in a laboratory". This paper will focus on evaluating the performance of Neural Collaborative Filtering and Gaussian Processes, as well as their multitask adaptations, on cancer datasets from CCLE, NCI60, GDSC and CTRP. These methods will be evaluated on model performance in regression prediction but also in interpretability.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2024

Enhancing Solar Power Forecasting with Regularized Constrained Quantile Regression Averaging and Bootstrapping Techniques

Probabilistic solar power forecasting (SPF) plays an essential role in optimizing power-grid operations by quantifying the forecast uncertainty. To improve the accuracy and robustness of probabilistic SPF, this paper introduces the regularized constrained quantile regression averaging (rCQRA) method to combine outputs from multiple PSPF models. In addition, a bootstrapping method was used to quantify model uncertainty, providing insights into the reliability and significance of each ensemble component. To evaluate its efficacy, the proposed rCQRA method is used to integrate four PSPF methods. The resulting SPF models are trained and validated using a real-world six-year dataset from a rooftop solar plant in the USA. The performance of the proposed rCQRA method is evaluated and compared with two benchmark methods under three categories of weather conditions. It is shown that the rCQRA method has superior performance in its forecast reliability, sharpness, and accuracy.

Ensemble learning, probabilistic solar power forec↗
NASA NTRSDate not supplied

Space Flown Rodent Liver RNA Sequencing Data for Machine Learning in Space Biology Research

High-throughput nucleic acid sequencing (DNA-seq, RNA-seq) has become widespread in biomedical research due to the growing availability and affordability of these assays. Data analysis has been accelerated in recent years by the adoption of artificial intelligence (AI) and machine learning (ML) techniques by biomedical researchers. In space biology research, RNAseq datasets from space-flown experimental samples are critical for characterizing the gene expression aberrations associated with exposure to spaceflight stressors. However, space biological experiments tend to be very low sample size, so identifying proper AI/ML algorithms for sequencing data analysis is an ongoing challenge since these algorithms typically require large sample size. The NASA Science Mission Directorate (SMD) has started the “Benchmark Initiative for AI/ML”, focused on creating datasets meant for three main applications: 1) scientific benchmarking, which finds the best algorithm for a specific problem; 2) application benchmarking, which measures algorithm performance against a set of parameters; and 3) system benchmarking, which evaluates performance of hardware and software architecture. These scientific benchmarks consist of an AI-ready dataset and a reference implementation on a specific scientific question. In this work, we focused on generating standardized datasets to allow the scientific community to benchmark AI/ML algorithms in the domain of space biology. We present here a standardized, AI-ready, publicly available benchmark dataset for space biology RNA-seq data as a collaboration between the NASA AI4LS (Artificial Intelligence for Life Sciences) working group. and NASA’s SMD. This dataset consists of space-flown and ground control mouse liver found in the NASA GeneLab omics database. However, to amplify the small sample number (n=112 samples) for ML purposes, we employ Gaussian noise and a generative adversarial network to extend this dataset to 6,000 synthetic samples, matching the original gene expression characteristics.

James Casaletto↗
DOE OSTI2022

Multimodal sensor fusion framework for residential building occupancy detection

For several years now, smart building energy systems have been a research area of intensive activity. In light of the increasing need for sustainable buildings and energy systems, this trend motivates an increasing need for a solution to reduce carbon dioxide emissions and improve energy efficiency. This work proposes a high-performing and transferable occupancy detection framework that combines sensor data from different data modalities, including time series environmental data (temperature, humidity, and illuminance), image data, and acoustic energy data using ensemble method. To draw out the best prediction performance in each modality, the proposed framework was developed, including various models that were designed to learn the occupancy patterns reflected in the physical data streams. To tackle the time series environmental data, we designed two variants of an occupancy detection spatiotemporal pattern network (Occ-STPN) that performs both feature level and decision level fusion, respectively. We also propose a new metric; the fading memory mean square error (FMMSE), that provides a fair evaluation and penalization of delayed occupancy predictions. Multiple open-sourced datasets, including the Electricity Consumption and Occupancy and the University of California, Irvine's (UCI) building occupancy detection dataset, along with our own real data collected from six different houses, were used to validate the algorithms' performance. The experimental results presented herein break down the performance for each sensing modality, and a detailed analysis of the performance is also discussed.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

Energy storage planning for enhanced resilience of power systems against wildfires and heatwaves

Extreme weather events pose significant risks to power grid stability due to their severe consequences and potential for widespread failures. Energy storage systems hold great potential for enhancing grid resilience against such events by providing reliable power during peak demand periods. However, accurately quantifying the size, location, and investment costs of new energy storage assets is a complex task, as energy storage planning decisions depend on the investment choices of other generation technologies and the integration of new transmission projects. Here, this paper presents a novel capacity expansion planning framework that simultaneously optimizes investments in energy storage, generation, and transmission, determining their optimal size, location, and type, while incorporating extreme weather events into long-term planning. More specifically, our stress-event-informed planning framework integrates the impact of heatwaves and wildfires into the planning process, identifying least-cost investment solutions that comply with policy goals and enhance grid resilience. The proposed framework employs machine-learning-based modeling to project heatwave-induced loads and performance-based risk assessment to evaluate wildfire-driven transmission line derates. Using industry-standard datasets to accurately represent the transmission topology of the Western Interconnection (WI) system, the proposed framework is applied to the WI 40-zone system, with investment decisions reported for the years 2030, 2035, and 2040. Simulation results reveal that with just a 10% increase in investment costs, resilience against extreme events can be significantly improved, with investment decisions heavily favoring energy storage, particularly 4-hour energy storage systems.

25 ENERGY STORAGE↗
DOE OSTI2023

Exact solution of burst pressure for thick-walled pipes using the flow theory of plasticity

This paper develops an exact solution of burst pressure for defect-free, thick-walled pipes using the flow theory of plasticity in terms of the average shear stress yield criterion (or Zhu-Leis yield criterion). The pipe steel is assumed to obey the power-law strain hardening rule, and large plastic deformation is described by the finite strain theory. On this basis, internal pressure is obtained as a power series function of the effective strains on the inside and outside surfaces of the thick-walled pipe with use of the second Bernoulli numbers. At burst failure, the Zhu-Leis flow solution of burst pressure is determined as a power series solution, and the burst effective strains, the burst effective stresses, and the burst pressure are functions of the diameter ratio (D o / D i ), strain hardening exponent (n), and ultimate tensile strength (UTS). Similarly, the Tresca and von Mises flow solutions of burst pressure are also determined as a power series solution in terms of the Tresca and von Mises yield criteria. A general closed-from exact solution of burst pressure is then proposed for the three yield criteria, and the results showed that the proposed exact solution matches well with the power series solution for each yield criterion. Moreover, the von Mises flow solution is an upper bound prediction, the Tresca flow solution is a lower bound prediction, and the Zhu-Leis flow solution is an intermediate prediction that agrees well with the finite element analysis results of burst pressure for thick-walled pipes. Furthermore, two datasets of full-scale burst tests are then utilized to evaluate and validate the proposed flow solutions of burst pressure for both thin and thick-walled pipes.

42 ENGINEERING↗
DOE OSTI2021

Revisiting Huffman Coding: Toward Extreme Performance on Modern GPU Architectures

Today's high-performance computing (HPC) applications are producing vast volumes of data, which are challenging to store and transfer efficiently during the execution, such that data compression is becoming a critical technique to mitigate the storage burden and data movement cost. Huffman coding is arguably the most efficient Entropy coding algorithm in information theory, such that it could be found as a fundamental step in many modern compression algorithms such as DEFLATE. On the other hand, today's HPC applications are more and more relying on the accelerators such as GPU on supercomputers, while Huffman encoding suffers from low throughput on GPUs, resulting in a significant bottleneck in the entire data processing. In this paper, we propose and implement an efficient Huffman encoding approach based on modern GPU architectures, which addresses two key challenges: (1) how to parallelize the entire Huffman encoding algorithm, including codebook construction, and (2) how to fully utilize the high memory-bandwidth feature of modern GPU architectures. The detailed contribution is fourfold. (1) We develop an efficient parallel codebook construction on GPUs that scales effectively with the number of input symbols. (2) We propose a novel reduction based encoding scheme that can efficiently merge the codewords on GPUs. (3) We optimize the overall GPU performance by leveraging the state-of-the-art CUDA APIs such as Cooperative Groups. (4) We evaluate our Huffman encoder thoroughly using six real-world application datasets on two advanced GPUs and compare with our implemented multi-threaded Huffman encoder. Experiments show that our solution can improve the encoding throughput by up to 5.0x and 6.8x on NVIDIA RTX 5000 and V100, respectively, over the state-of-the-art GPU Huffman encoder, and by up to 3.3x over the multi-thread encoder on two 28-core Xeon Platinum 8280 CPUs.

Tian, Jiannan↗