Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “network performance evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 307 records · Page 17
All sourcesNASA NTRSDOE OSTI
DOE OSTI2021

Q-VR: System-Level Design for Future Mobile Collaborative Virtual Reality

High Quality Mobile Virtual Reality (VR) is what the incoming graphics technology era demands: users around the world, regardless of their hardware and network conditions, can all enjoy the immersive virtual experience. However, the state-of-the-art software-based mobile VR designs cannot fully satisfy the realtime performance requirements due to the highly interactive nature of user's actions and complex environmental constraints during VR execution. Inspired by the unique human visual system effects and the strong correlation between VR motion features and realtime hardware-level information, we propose Q-VR, a novel dynamic collaborative rendering solution via software-hardware co-design for enabling future low-latency high-quality mobile VR. At software-level, Q-VR provides flexible high-level tuning interface to reduce network latency while maintaining user perception. At hardware-level, Q-VR accommodates a wide spectrum of hardware and network conditions across users by effectively leveraging the computing capability of the increasingly powerful VR hardware. Extensive evaluation on real-world games demonstrates that Q-VR can achieve an average end-to-end performance speedup of 3.4x (up to 6.7x) over the traditional local rendering design in commercial VR devices, and a 4.1x frame rate improvement over the state-of-the-art static collaborative rendering.

Xie, Chenhao↗
DOE OSTI2022

Selecting Post-Processing Schemes for Accurate Detection of Small Objects in Low-Resolution Wide-Area Aerial Imagery

In low-resolution wide-area aerial imagery, object detection algorithms are categorized as feature extraction and machine learning approaches, where the former often requires a post-processing scheme to reduce false detections and the latter demands multi-stage learning followed by post-processing. In this paper, we present an approach on how to select post-processing schemes for aerial object detection. We evaluated combinations of each of ten vehicle detection algorithms with any of seven post-processing schemes, where the best three schemes for each algorithm were determined using average F-score metric. The performance improvement is quantified using basic information retrieval metrics as well as the classification of events, activities and relationships (CLEAR) metrics. We also implemented a two-stage learning algorithm using a hundred-layer densely connected convolutional neural network for small object detection and evaluated its degree of improvement when combined with the various post-processing schemes. The highest average F-scores after post-processing are 0.902, 0.704 and 0.891 for the Tucson, Phoenix and online VEDAI datasets, respectively. The combined results prove that our enhanced three-stage post-processing scheme achieves a mean average precision (mAP) of 63.9% for feature extraction methods and 82.8% for the machine learning approach.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Quantum Network Repeater Simulation Package (QNPack) v1.0.0

Existing quantum network testbeds and prototypes are typically implemented in laboratory experiments with limited functionality and operate in a tightly coupled manner. These characteristics make it difficult to evaluate the behavior of emerging quantum platforms and protocols beyond local resource limitations. The QNPack software enables full-stack modeling of quantum repeater networks that allows researchers to simulate new approaches, and understand emergent behaviors, from local to national scales. Novel capabilities include: (1) performant and accurate quantum models that can be executed as independent simulations, which will allow users to characterize specific analog quantum processes, quantum devices, and protocols, (2) a modular and extensible approach to the simulation architecture to accommodate new quantum technologies, protocols, and topologies as they are developed across the quantum repeater generations under evaluation, and (3) a focus on managing the interactions between the simulation framework sub-components so that they may be re-used and composed in flexible ways.

Kissel, Ezra [Lawrence Berkeley National Laborator↗
DOE OSTI2025

Evaluating Function-as-a-Service (FaaS) frameworks for the Accelerator Control System

As particle accelerator control systems evolve in complexity and scale, the need for responsive, scalable, and cost-effective computational infrastructure becomes increasingly critical. Function-as-a-Service (FaaS) offers an alternative to traditional monolithic architecture by enabling event-driven execution, automatic scaling, and fine-grained resource utilization. This paper explores the applicability and performance of FaaS frameworks in the context of a modern particle accelerator control system, with the objective of evaluating their suitability for short lived and triggered workloads. In this paper, we evaluate prominent open-source FaaS platforms in executing functional logic, triggers, and diagnostics routines. Evaluation metrics consist of cold-start latency, scalability, performance, integration with other open-source tools like Kafka. Experimental workloads were designed to simulate real-world control tasks when implemented as stateless FaaS functions. These workloads were benchmarked under various invocation loads and network conditions. Self-hosted FaaS platforms, when deployed within accelerator networks, offer greater control over execution environment, better integration with legacy systems, and support for real-time guarantees when paired with message queues. Based on lessons learned and evaluation metrics, this paper describes reliability of the FaaS framework for the Accelerator Control Systems (ACS).

Jaikar, A. [Fermilab] (ORCID:0000000332046217)↗
DOE OSTI2021

Fixed-angle conjectures for the quantum approximate optimization algorithm on regular MaxCut graphs

The quantum approximate optimization algorithm (QAOA) is a near-term combinatorial optimization algorithm suitable for noisy quantum devices. However, little is known about performance guarantees for p > 2. A recent work computing MaxCut performance guarantees for 3-regular graphs conjectures that any d-regular graph evaluated at particular fixed angles has an approximation ratio greater than some worst-case guarantee. In this work, we provide numerical evidence for this fixed angle conjecture for p < 12. We compute and provide these angles via numerical optimization and tensor networks. These fixed angles serve for an optimization-free version of QAOA and have universally good performance on any 3-regular graph. Heuristic evidence is presented for the fixed angle conjecture on graph ensembles, which suggests that these fixed angles are "close" to global optimum. Under the fixed angle conjecture, QAOA has a larger performance guarantee than the Goemans Williamson algorithm on 3-regular graphs for p >= 11.

Wurtz, Jonathan↗
DOE OSTI2022

Improving the particle dry deposition scheme in the CMAQ photochemical modeling system

Dry deposition of atmospheric aerosols in large-scale models is a critical, but highly uncertain, sink process with a strong dependence on particle size, meteorological conditions, and land surface properties. This study investigates the particle dry deposition scheme implemented in the standard Community Multiscale Air Quality (CMAQ) model v5.2.1, characterizes its underlying parameterized components with comparison to a similar scheme in a contemporary regional-scale model, and proposes two updated schemes that are then evaluated with available ambient particle deposition velocity (V d ) measurements. Both updated schemes reduce the surprisingly strong dependence of deposition velocity on the aerosol mode width, with one scheme further introducing a dependence on vegetation coverage that is broadly consistent with variability in observations between vegetated and non-vegetated surfaces. Compared to the base scheme, the updated scheme with vegetation dependence increases V d for submicron particles and decreases it for larger particles by an average of 37% and –66%, respectively. This scheme performs statistically better than the base scheme, reducing fractional biases by 56%–97% for vegetated land-use types and has roughly equivalent performance over water. The base and updated schemes are tested with three annual CMAQ (v5.2.1) simulations for the year 2011; predicted ambient aerosol concentrations are evaluated with routine monitoring network observations and predicted dry deposition fluxes are evaluated with data from the Clean Air Status and Trends Network (CASTNET). The updated scheme with vegetation dependence reduces negative fractional biases for PM 10 by 41% and positive fractional biases for PM 2.5 organic carbon by 15%. This scheme has been incorporated into the most recent publicly accessible versions of CMAQ (v5.3 and beyond) to replace the scheme used in previous versions of CMAQ (v4.5 through v5.2.1).

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

PreMevE Update: Forecasting Ultra-Relativistic Electrons Inside Earth's Outer Radiation Belt

Energetic electrons inside Earth's Van Allen belts pose a major radiation threat to spaceborne electronics that often play vital roles in modern society. Ultra-relativistic electrons with energies greater than or equal to two megaelectron-volt (MeV) are of particular interest, and thus forecasting these ≥2 MeV electrons has a significant meaning to all space sectors. Here, we update the latest development of the predictive model for MeV electrons in the outer radiation belt. The new version, called PREdictive MEV Electron (PreMevE)-2E, forecasts ultra-relativistic electron flux distributions across the outer belt, with no need for in situ measurements of the trapped MeV electron population except at the geosynchronous orbit (GEO). Model inputs include precipitating electrons observed in low-Earthorbits by NOAA satellites, upstream solar wind speeds and densities from solar wind monitors, as well as ultra-relativistic electrons measured by one Los Alamos GEO satellite. We evaluated 32 supervised machine learning models that fall into four different classes of linear and neural network architectures, and successfully tested ensemble forecasting by using groups of top-performing models. All models are individually trained, validated, and tested by in situ electron data from NASA's Van Allen Probes mission. It is shown that the final ensemble model outperforms individual models at most L-shells, and this PreMevE-2E model can provide 25-h (~1-day) and 50-h (~2-day) forecasts with high mean performance efficiency and correlation values. Our results also suggest that this new model is dominated by nonlinear components at L-shells <~4 for ultra-relativistic electrons, different from the dominance of linear components for 1 MeV electrons as previously discovered.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2026

Comprehensive framework for assessing and optimizing existing research networks

Conservation, monitoring, and research networks, or collections of ecological research sites unified under a common mission of data collection or a research mission, are essential infrastructure for understanding large landscapes. However, most networks developed opportunistically over decades rather than through systematic design, creating potential limitations in the ability to address conservation challenges across entire regions. We developed a framework to evaluate how well an existing research network represents the environmental conditions its members study and devised an approach to rank sites of priority for strategic expansion. Our approach measures performance through environmental representativeness, geographic coverage, and adequacy for scientific inference and thus optimizes limited monitoring resources to maximize scientific impact. We demonstrated this approach with the U.S. Department of Agriculture (USDA) Forest Service Experimental Forests and Ranges Network (EFRN), a 79‐site network across the United States that grew opportunistically over a century. At the national scale, the network effectively captured high‐biomass forests important for carbon cycle research; 82% of forest biomass was in well‐represented areas. Some areas in Texas, Florida, the Rocky Mountains, and the West Coast had no relevant EFRN sites, which limits the ability to make regional inferences. A fundamental challenge for the EFRN was that sites improving regional extent coverage sometimes provided minimal national benefits, which can create conflicts between local and global priorities. Adding the highest‐ranked candidate site provided a relevant site for 17% of currently poorly represented 1‐km pixel cells nationally, but regional and national site rankings varied considerably due to nested spatial inference. This framework provides quantitative tools for strategic infrastructure decision‐making, ensures that limited monitoring resources maximize conservation impact, and can be applied broadly to address the widespread challenge of optimizing conservation and monitoring networks worldwide.

additional site↗
DOE OSTI2021

Developing a Hybrid Electric Vehicle Eco-Cooperative Adaptive Cruise Control System at Signalized Intersections.

This study develops an eco-driving strategy for hybrid electric vehicles (HEVs) in the vicinity of signalized intersections, entitled HEV Eco-Cooperative Adaptive Cruise Control at Intersections (Eco-CACC-I). The proposed system computes real-time, energy-optimized vehicle trajectories using HEV vehicle dynamics and energy consumption models. In the proposed system, a simple HEV energy model is used to compute the instantaneous fuel consumption. This HEV energy model is selected since it is general, transferable, and can be easily used to compute instantaneous energy consumption levels for HEVs without the additional input of vehicle engine data or complicated power control strategies. In addition, a vehicle dynamics model is used to capture the relationship between speed, acceleration level, and tractive/resistance forces on vehicles. The energy-optimum problem is formulated as an optimization problem with constraints, which is solved using a moving-horizon dynamic programming approach. The proposed HEV Eco-CACC-I system was tested to evaluate its performance for various speed limits, roadway grades, and signal timings. Lastly, the proposed HEV controller was implemented in a microscopic traffic simulation software to test its network-wide performance. The test results from an arterial corridor with three signalized intersections demonstrate that the proposed system can effectively reduce stop-and-go traffic in the vicinity of signalized intersections producing savings of 7.4% in energy consumption, 5.8% in traffic delay and 23% vehicle stops, respectively.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2025

Probabilistic Resilience-Oriented Assessment Approach for Transmission Networks Under Wildfires

The rising threat of wildfires poses significant challenges to power transmission networks, particularly in areas prone to such disasters. Traditional approaches for wildfire risk assessment neglect some potential wildfire scenarios. Here, this paper introduces a probabilistic resilience-oriented assessment approach for power transmission networks to address this gap. Initially, a probabilistic wildfire model is developed to capture uncertainties in ignition, intensity, and fire spread. Next, a spatiotemporal fragility model is constructed to assess the impact of wildfires on transmission corridors, incorporating Thermal Aging (TA) and Dynamic Thermal Rate (DTR) change. Finally, a comprehensive resilience metric is defined to evaluate system performance, leveraging the fragility model to determine component and system-level resilience. The approach employs a combinatorial enumeration method to generate potential wildfire scenarios, enhanced by an impact-increment-based state enumeration (IISE) method for computational efficiency. The proposed method provides critical insights for identifying system vulnerabilities and developing robust strategies to protect transmission networks from wildfires. The efficacy of this approach is validated through extensive scenarios of the RTS-GMLC system across Southern California, Nevada and Arizona.

Vahedi, Soroush [Univ. of Connecticut, Storrs, CT ↗
DOE OSTI2020

A priori analysis on deep learning of subgrid-scale parameterizations for Kraichnan turbulence

In the present study, we investigate different data-driven parameterizations for large eddy simulation of two-dimensional turbulence in the a priori settings. These models utilize resolved flow field variables on the coarser grid to estimate the subgrid-scale stresses. We use data-driven closure models based on localized learning that employs a multilayer feedforward artificial neural network with point-to-point mapping and neighboring stencil data mapping, and convolutional neural network fed by data snapshots of the whole domain. The performance of these data-driven closure models is measured through a probability density function and is compared with the dynamic Smagorinksy model (DSM). The quantitative performance is evaluated using the cross-correlation coefficient between the true and predicted stresses. We analyze different frameworks in terms of the amount of training data, selection of input and output features, their characteristics in modeling with accuracy, and training and deployment computational time. We also demonstrate computational gain that can be achieved using the intelligent eddy viscosity model that learns eddy viscosity computed by the DSM instead of subgrid-scale stresses. Here, we detail the hyperparameters optimization of these models using the grid search algorithm.

42 ENGINEERING↗
DOE OSTI2023

Field programmable spin arrays for scalable quantum repeaters

The large scale control over thousands of quantum emitters desired by quantum network technology is limited by the power consumption and cross-talk inherent in current microwave techniques. Here we propose a quantum repeater architecture based on densely-packed diamond color centers (CCs) in a programmable electrode array, with quantum gates driven by electric or strain fields. This ‘field programmable spin array’ (FPSA) enables high-speed spin control of individual CCs with low cross-talk and power dissipation. Integrated in a slow-light waveguide for efficient optical coupling, the FPSA serves as a quantum interface for optically-mediated entanglement. We evaluate the performance of the FPSA architecture in comparison to a routing-tree design and show an increased entanglement generation rate scaling into the thousand-qubit regime. Our results enable high fidelity control of dense quantum emitter arrays for scalable networking.

42 ENGINEERING↗
DOE OSTI2024

Path-BigBird: An AI-Driven Transformer Approach to Classification of Cancer Pathology Reports

PURPOSE Surgical pathology reports are critical for cancer diagnosis and management. To accurately extract information about tumor characteristics from pathology reports in near real time, we explore the impact of using domain-specific transformer models that understand cancer pathology reports. METHODS We built a pathology transformer model, Path-BigBird, by using 2.7 million pathology reports from six SEER cancer registries. We then compare different variations of Path-BigBird with two less computationally intensive methods: Hierarchical Self-Attention Network (HiSAN) classification model and an offthe-shelf clinical transformer model (Clinical BigBird). We use five pathology information extraction tasks for evaluation: site, subsite, laterality, histology, and behavior. Model performance is evaluated by using macro and micro F 1 scores. RESULTS We found that Path-BigBird and Clinical BigBird outperformed the HiSAN in all tasks. Clinical BigBird performed better on the site and laterality tasks. Versions of the Path-BigBird model performed best on the two most difficult tasks: subsite (micro F 1 score of 72.53, macro F 1 score of 35.76) and histology (micro F 1 score of 80.96, macro F 1 score of 37.94). The largest performance gains over the HiSAN model were for histology, for which a Path-BigBird model increased the micro F 1 score by 1.44 points and the macro F 1 score by 3.55 points. Overall, the results suggest that a Path-BigBird model with a vocabulary derived from wellcurated and deidentified data is the best-performing model. CONCLUSION The Path-BigBird pathology transformer model improves automated information extraction from pathology reports. Although Path-BigBird outperforms Clinical BigBird and HiSAN, these less computationally expensive models still have utility when resources are constrained.

60 APPLIED LIFE SCIENCES↗
DOE OSTI2026

Neural Network‐Based Methods for Ocean Surface Wave Measurement Using Submarine Distributed Acoustic Sensing (DAS)

Two new data-driven models for estimating ocean surface waves from distributed acoustic sensing (DAS) submarine cable strain rate are developed using supervised machine learning on a 10-day data set collected offshore of Oliktok Point, Alaska. The new models were trained on target data from seafloor pressure moorings at three sites spaced evenly along 27.1 km of cable and were benchmarked against an empirical transfer function method previously used to estimate waves from DAS. A model which uses convolutional neural networks to transform 2-km frequency-wavenumber strain spectra to seafloor pressure spectra outperforms the benchmark in wave height prediction (RMSE of 0.15 vs. 0.41 m) and period prediction (0.29 vs. 0.37 s) when evaluated on a held-out test data set. When applied to a DAS data set collected on the same cable 2 years prior, the CNN-based model maintained similar significant wave height performance (RMSE = 0.23 m) relative to available satellite altimetry data. A two-hidden-layer, fully connected neural network which transforms 1-D strain spectra to seafloor pressure spectra also outperforms the benchmark in wave height prediction (RMSE of 0.19 vs. 0.41 m), but does not generalize as well to the prior data. Regression-based machine learning is useful for estimating waves from DAS data when the pressure-strain relationship varies temporally and spatially across different wave conditions. Models can be applied to DAS data to measure waves with higher spatial resolution and longer temporal coverage than traditional methods, which often measure waves only at a single point.

Davis, Jacob R. [Univ. of Washington, Seattle, WA ↗
DOE OSTI2025

IoT-based retrofit information diffusion in future smart communities

Community-scale building retrofits are not merely scaled-up versions of single-building retrofits. They involve complex challenges, such as reconciling individual interests with collective goals and managing the dynamic interplay between buildings through mechanisms like power grids and social connections. Internet of Things (IoT) connectivity holds the potential to leverage these interplays to balance individual and collective interests effectively in smart communities. One critical aspect of this interplay is information diffusion, which shapes how retrofit decisions spread among neighbors, influencing individual choices and ultimately impacting community-level retrofit outcomes. In other words, IoT-based smart devices automatically push tailored retrofit notifications to homeowners, which completely changes the format of information diffusion in the future. To investigate this influence by such information diffusion, the study used CityBES to simulate energy performance for different retrofits and applied an information diffusion model to analyze how decisions spread in a networked community of 192 buildings. The diffusion process was modeled on a weighted, directed network, capturing the dynamics of information flow and decision-making across 16 scenarios. Individual retrofit benefits were evaluated through payback years, while community-level retrofit outcomes were assessed using greenhouse gas (GHG) emission reductions. The results demonstrate that easier information diffusion among neighbors encourages households to prioritize retrofit measures that align with the majority’s optimal choices, even at the expense of individual financial benefits. In this case, such collective prioritization enhanced community-level retrofit performance, increasing GHG emission reductions by up to 29.4 %. However, this improvement came with trade-offs, as the average payback period for households extended by approximately 1.74 years. These findings highlight the potential of IoT-based information diffusion in future smart communities to coordinate individual interests with collective goals, ultimately accelerating community-level building retrofits.

Shu, Lei↗
DOE OSTI2025

Initial Mobility Analysis for ORNL VA-EDH Synthetic Populations

Travel burdens are a major barrier to healthcare access among US Veteran patient populations, particularly those residing in rural areas. Spatial accessibility to points of care for US Veteran populations is commonly assessed in two ways. The first approach uses open data from the US Census to represent collective travel burdens, for example the distance between population-weighted census tract centroids and VHA points of care. The second approach uses restricted-access VHA patient data to measure travel costs (e.g., distance, time) for accessing points of care with respect to geolocated patient addresses and real or approximated transportation networks. While the advantage of the open data approach lies in its reproducibility, it has notable limitations in its tendency to infer individual travel behavior from aggregate population characteristics, a problem known as ecological fallacy. Conversely, while the patient data approach is able to account for individual travel behavior, its ability to account for localized access disparities (e.g., a neighborhood with exceptionally high transportation costs) and patient demographics is limited as protecting individual patient data requires their storage in closed systems with limited capacity for adequately modeling real-world travel patterns or for supplementing patient attributes. Additionally, the patient data approach cannot account for veterans who are not enrolled in the VHA system but who may be eligible for care. These challenges limit the ability to perform “what if” analyses on the effects of place-specific interventions on veteran populations with high access barriers to healthcare. To address these challenges, we explore the application of realistic synthetic populations to examine travel burdens and spatial accessibility issues among veteran patient populations. Synthetic populations provide a virtual, individually-resolved and cross-sectional representation of the veteran patient population that enables investigation of spatial access to points of care in ways in which aggregate data and patient data do not. First, synthetic populations allow one to directly assess how individuals access points of care, from synthesized residential locations to outpatient facilities on real-world transportation networks. Modeling access to points of care at the individual scale addresses the ecological fallacy problem associated with using aggregated census data to represent veteran populations and patterns of movement. Second, synthetic populations provide a means of completely representing an area’s veteran population using only publicly available, anonymized census microdata from the American Community Survey (ACS) to ensure the privacy of real-world individuals. Generating synthetic populations from the ACS also expands descriptive characteristics beyond what patient data typically offers to include socio-demographic, economic, housing, and mobility attributes. More detailed profiles of both VHA patient populations and veterans not enrolled in the VA system will provide a comprehensive picture of groups that may benefit from interventions or outreach. As an initial exercise for using synthetic populations to measure veteran travel burdens to VA care, we apply Oak Ridge National Laboratory’s (ORNL) UrbanPop capability to generate a series of synthetic VHA patient populations for 9 Veterans Integrated Services Networks (VISN) market areas in 9 Census Divisions across the continental United States, which are listed in Table 1. We use UrbanPop to produce synthetic populations for the VISN markets selected for each US Census Division, then assign VA outpatient clinic destinations to synthetic VHA patients based on travel about each VISN market’s road network. To demonstrate using the synthetic populations to evaluate healthcare travel burdens, we compare the time-based impedance between simulated home locations and VA outpatient clinics in each VISN market. We then perform validation exercises on the synthetic populations with respect to neighborhood (block group) demographic composition as well as patient mobility, comparing aggregate origin-destination statistics for the synthetic population to outpatient visits available in restricted patient data from the VA’s Corporate Data Warehouse (CDW) database.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Let’s Unleash the Network Judgment: A Self-Supervised Approach for Cloud Image Analysis

Accurate cloud type identification and coverage analysis are crucial in understanding the Earth’s radiative budget. Traditional computer vision methods rely on low-level visual features of clouds for estimating cloud coverage or sky conditions. Several handcrafted approaches have been proposed; however, scope for improvement still exists. Newer deep neural networks (DNNs) have demonstrated superior performance for cloud segmentation and categorization. These methods, however, need expert engineering intervention in the preprocessing steps—in the traditional methods—or human assistance in assigning cloud or clear sky labels to a pixel for training DNNs. Such human mediation imposes considerable time and labor costs. We present the application of a new self-supervised learning approach to autonomously extract relevant features from sky images captured by ground-based cameras, for the classification and segmentation of clouds. We evaluate a joint embedding architecture that uses self-knowledge distillation plus regularization. We use two datasets to demonstrate the network’s ability to classify and segment sky images—one with ~85,000 images collected from our ground-based camera and another with 400 labeled images from the WSISEG database. We find that this approach can discriminate full-sky images based on cloud coverage, diurnal variation, and cloud base height. Additionally, it semantically segments the cloud areas without labels. The approach shows competitive performance in all tested tasks, suggesting a new alternative for cloud characterization.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Optimizing High Performance Markov Clustering for Pre-Exascale Architectures

HipMCL is a high-performance distributed memory implementation of the popular Markov Cluster Algorithm (MCL) and can cluster large-scale networks within hours using a few thousand CPU-equipped nodes. It relies on sparse matrix computations and heavily makes use of the sparse matrix-sparse matrix multiplication kernel (SpGEMM). The existing parallel algorithms in HipMCL are not scalable to Exascale architectures, both due to their communication costs dominating the runtime at large concurrencies and also due to their inability to take advantage of accelerators that are increasingly popular. In this work, we systematically remove scalability and performance bottlenecks of HipMCL. We enable GPUs by performing the expensive expansion phase of the MCL algorithm on GPU. Additionally, we propose a CPU-GPU joint distributed SpGEMM algorithm called pipelined Sparse SUMMA and integrate a probabilistic memory requirement estimator that is fast and accurate. Furthermore, we develop a new merging algorithm for the incremental processing of partial results produced by the GPUs, which improves the overlap efficiency and the peak memory usage. We also integrate a recent and faster algorithm for performing SpGEMM on CPUs. We validate our new algorithms and optimizations with extensive evaluations. With the enabling of the GPUs and integration of new algorithms, HipMCL is up to 12.4x faster, being able to cluster a network with 70 million proteins and 68 billion connections just under 15 minutes using 1024 nodes of ORNL's Summit supercomputer.

97 MATHEMATICS AND COMPUTING↗