Fast multiplication of random dense matrices with sparse matrices
This work focuses on accelerating the multiplication of a dense random matrix with a (fixed) sparse matrix, which is frequently used in sketching algorithms.
Demmel, James↗
SEARCH · Engineering Papers
Results for “sparse matrices”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Explicit Quantum Circuits for Block Encodings of Certain Sparse Matrices
Many standard linear algebra problems can be solved on a quantum computer by using recently developed quantum linear algebra algorithms that make use of block encodings and quantum eigenvalue/singular value transformations. A block encoding embeds a properly scaled matrix of interest A in a larger unitary transformation U that can be decomposed into a product of simpler unitaries and implemented efficiently on a quantum computer. Although quantum algorithms can potentially achieve exponential speedup in solving linear algebra problems compared to the best classical algorithm, such a gain in efficiency ultimately hinges on our ability to construct an efficient quantum circuit for the block encoding of A, which is difficult in general, and not trivial even for well structured sparse matrices. Here, in this paper, we give a few examples on how efficient quantum circuits can be explicitly constructed for some well structured sparse matrices and discuss a few strategies used in these constructions. We also provide implementations of these quantum circuits in MATLAB.
97 MATHEMATICS AND COMPUTING↗
Distributed Many-to-Many Protein Sequence Alignment using Sparse Matrices
Identifying similar protein sequences is a core step in many computational biology pipelines such as detection of homologous protein sequences, generation of similarity protein graphs for downstream analysis, functional annotation, and gene location. Performance and scalability of protein similarity search have proven to be a bottleneck in many bioinformatics pipelines due to increase in cheap and abundant sequencing data. This work presents a new distributed-memory software PASTIS. PASTIS relies on sparse matrix computations for efficient identification of possibly similar proteins. We use distributed sparse matrices for scalability and show that the sparse matrix infrastructure is a great fit for protein similarity search when coupled with a fully-distributed dictionary of sequences that allow remote sequence requests to be fulfilled. Our algorithm incorporates the unique bias in amino acid sequence substitution in search without altering basic sparse matrix model, and in turn, achieves ideal scaling up to millions of protein sequences.
97 MATHEMATICS AND COMPUTING↗
parGeMSLR: A parallel multilevel Schur complement low-rank preconditioning and solution package for general sparse matrices
Not Available
Xu, Tianshi (ORCID:0000000331191957)↗
Fast Low-Rank QRCP of Sparse Matrices -- with RandBLAS!
Explore the source record for details and available documents.
Murray, Riley John↗
SparseLU, A Novel Algorithm and Math Library for Sparse LU Factorization
Decomposing sparse matrices into lower and upper triangular matrices (sparse LU factorization) is a key operation in many computational scientific applications. We developed SparseLU, a sparse linear algebra library that implements a new algorithm for LU factorization on general sparse matrices. The new algorithm divides the input matrix into tiles to which OpenMP tasks are created for factorization computation, where only tiles that contain nonzero elements are computed. For comparative performance analysis, we used the reference library SuperLU. Testing was performed on synthetically generated matrices which replicate the conditions of the real-world matrices. SparseLU is able to reach a mean speedup of ~29× compared to SuperLU.
Valero Lara, Pedro↗
Fast truncated SVD of sparse and dense matrices on graphics processors
We investigate the solution of low-rank matrix approximation problems using the truncated singular value decomposition (SVD). For this purpose, we develop and optimize graphics processing unit (GPU) implementations for the randomized SVD and a blocked variant of the Lanczos approach. Our work takes advantage of the fact that the two methods are composed of very similar linear algebra building blocks, which can be assembled using numerical kernels from existing high-performance linear algebra libraries. Furthermore, the experiments with several sparse matrices arising in representative real-world applications and synthetic dense test matrices reveal a performance advantage of the block Lanczos algorithm when targeting the same approximation accuracy.
Computer Science↗
Reducing Communication in Graph Neural Network Training
Graph Neural Networks (GNNs) are powerful and flexible neural networks that use the naturally sparse connectivity information of the data. GNNs represent this connectivity as sparse matrices, which have lower arithmetic intensity and thus higher communication costs compared to dense matrices, making GNNs harder to scale to high concurrencies than convolutional or fully-connected neural networks. Here, we introduce a family of parallel algorithms for training GNNs and show that they can asymptotically reduce communication compared to previous parallel GNN training methods. We implement these algorithms, which are based on 1D, 1. 5D, 2D, and 3D sparse-dense matrix multiplication, using torch.distributed on GPU-equipped clusters. Our algorithms optimize communication across the full GNN training pipeline. We train GNNs on over a hundred GPUs on multiple datasets, including a protein network with over a billion edges.
97 MATHEMATICS AND COMPUTING↗
Accelerating GNNs on GPU Sparse Tensor Cores through N:M Sparsity-Oriented Graph Reordering
Recent advancements in GPU hardware support have introduced the capability to leverage N:M sparse patterns for substantial performance gains. Graphs in Graph Neural Networks (GNNs) are typically sparse, but the sparsity is often irregular, not conforming to such sparse patterns. In this paper, we propose a novel graph reordering algorithm, the first of its kind, to reshape irregular graph data into the N:M structured sparse pattern at the tile level, allowing linear-algebra-based graph operations in GNNs to benefit from the N:M sparse hardware. The optimization is lossless, maintaining the accuracy of GNN. It can remove 98-100\% violations of the N:M sparse patterns at the vector level, and increase the proportion of conforming graphs in SuiteSparse collection from 5-9\% to 88.7-93.5\%. On A100 GPUs, the optimization accelerates Sparse Matrix Matrix (SpMM) by up to 43X (2.3X -- 7.5X on average) and speeds up the key graph operations in GNNs on real graphs by as much as 8.6X (3.5X on average).
artificial intelligence, graph neural networks↗
Dynamics of disordered mechanical systems with large connectivity, free probability theory, and quasi-Hermitian random matrices
Disordered mechanical systems with high connectivity represent a limit opposite to the more familiar case of disordered crystals. Individual ions in a crystal are subjected essentially to nearest-neighbor interactions. In contrast, the systems studied in this paper have all their degrees of freedom coupled to each other. Thus, the problem of linearized small oscillations of such systems involves two full positive-definite and non-commuting matrices, as opposed to the sparse matrices associated with disordered crystals. Consequently, the familiar methods for determining the averaged vibrational spectra of disordered crystals, introduced many years ago by Dyson and Schmidt, are inapplicable for highly connected disordered systems. In this paper we apply random matrix theory (RMT) to calculate the averaged vibrational spectra of such systems, in the limit of infinitely large system size. At the heart of our analysis lies a calculation of the average spectrum of the product of two positive definite random matrices by means of free probability theory techniques. We also show that this problem is intimately related with quasi-hermitian random matrix theory (QHRMT), which means that the ‘hamiltonian’ matrix is hermitian with respect to a non-trivial metric. This extends ordinary hermitian matrices, for which the metric is simply the unit matrix. The analytical results we obtain for the spectrum agree well with our numerical results. The latter also exhibit oscillations at the high-frequency band edge, which fit well the Airy kernel pattern. We also compute inverse participation ratios of the corresponding amplitude eigenvectors and demonstrate that they are all extended, in contrast with conventional disordered crystals. Finally, we compute the thermodynamic properties of the system from its spectrum of vibrations. In addition to matrix model analysis, we also study the vibrational spectra of various multi-segmented disordered pendula, as concrete realizations of highly connected mechanical systems. A universal feature of the density of vibration modes, common to both pendula and the matrix model, is that it tends to a non-zero constant at vanishing frequency.
71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
Communication-Avoiding and Memory-Constrained Sparse Matrix-Matrix Multiplication at Extreme Scale
Sparse matrix-matrix multiplication (SpGEMM) is a widely used kernel in various graph, scientific computing and machine learning algorithms. In this paper, we consider SpGEMMs performed on hundreds of thousands of processors generating trillions of nonzeros in the output matrix. Distributed SpGEMM at this extreme scale faces two key challenges: (1) high communication cost and (2) inadequate memory to generate the output. Furthermore, we address these challenges with an integrated communication-avoiding and memory-constrained SpGEMM algorithm that scales to 262,144 cores (more than 1 million hardware threads) and can multiply sparse matrices of any size as long as inputs and a fraction of output fit in the aggregated memory. As we go from 16,384 cores to 262,144 cores on a Cray XC40 supercomputer, the new SpGEMM algorithm runs 10x faster when multiplying large-scale protein-similarity matrices.
97 MATHEMATICS AND COMPUTING↗
Quantum block encoding for one-pair semiseparable matrices
Quantum block encoding (QBE) is a crucial step in the development of most quantum algorithms, as it provides an embedding of a given matrix into a suitable larger unitary matrix. Historically, the development of efficient techniques for QBE has mostly focused on sparse matrices; less effort has been devoted to data-sparse (e.g., rank-structured) matrices. In this work we examine a particular case of rank structure, namely, one-pair semiseparable matrices. We present a new block encoding approach that relies on a suitable factorization of the given matrix as the product of triangular and diagonal factors. To encode the matrix, the algorithm needs $2\log(N)+7$ ancillary qubits. Assuming that the data input oracles can be implemented with polylogarithmic depth, or that a QRAM input model is available, our proposed method requires $\mathcal{O}({\rm polylog} (N))$ time and has an error of $\mathcal{O}(N^2)$, where $N$ is the matrix size.
Antonioli, Giacomo [Pisa U.; CERN] (ORCID:00090000↗
Feeder Power Disaggregation: A Data-Efficient Matrix Completion Approach: Preprint
This paper presents a data-driven algorithm for the feeder power disaggregation problem in distribution systems. Leveraging spatio-temporal power patterns in residential homes, residential power is discomposed into three components: sparse-switching loads, periodic loads, and photovoltaic generation, using two sparse matrices and a rank-one matrix. The matrix completion process is data-efficient because of the matrix sparsity and low rankness, along with the use of power system models. The proposed approach is tested using real-world residential datasets on a 33-bus distribution system, demonstrating accurate power disaggregation with efficient matrix completion.
distribution system↗
Feeder Power Disaggregation: A Data-Efficient Matrix Completion Approach
This paper presents a data-driven algorithm for the feeder power disaggregation problem in distribution systems. Leveraging spatio-temporal power patterns in residential homes, residential power is discomposed into three components: sparse-switching loads, periodic loads, and photovoltaic (PV) generation, which are characterized through the design of two sparse matrices and a low-rank matrix. The matrix completion process is data-efficient because of the matrix sparsity and low rankness, along with the use of power system models. The proposed approach is tested using real-world residential data set on a 33-bus distribution system, demonstrating accurate power disaggregation with efficient matrix completion.
distribution system↗
The Kokkos OpenMPTarget Backend: Implementation and Lessons Learned
As the supercomputing landscape diversifies, solutions such as Kokkos to write vendor agnostic applications and libraries have risen in popularity. Kokkos provides a programming model designed for performance portability, which allows developers to write a single source implementation that can run efficiently on various architectures. At its heart, Kokkos maps parallel algorithms to architecture and vendor specific backends written in lower level programming models such as CUDA and HIP. Another approach to writing vendor agnostic parallel code is using OpenMP’s directives based approach, which lets developers annotate code to express parallelism. It is implemented at the compiler level and is supported by all major high performance computing vendors, as well as the primary Open Source toolchains GNU and LLVM. Since its inception, Kokkos has used OpenMP to parallelize on CPU architectures. In this paper, we explore leveraging OpenMP for a GPU backend and discuss the challenges we encountered when mapping the Kokkos APIs and semantics to OpenMP target constructs. As an exemplar workload we chose a simple conjugate gradient solver for sparse matrices. We find that performance on NVIDIA and AMD GPUs varies widely based on details of the implementation strategy and the chosen compiler. Furthermore, the performance of the OpenMP implementations decreases with increasing complexity of the investigated algorithms.
Gayatri, Rahulkumar↗
A higher-order finite-element implementation of the nonlinear Fokker–Planck collision operator for charged particle collisions in a low density plasma
Collisions between particles in a low density plasma are described by the Fokker–Planck collision operator. In applications, this nonlinear integro-differential operator is often approximated by linearised or ad-hoc model operators due to computational cost and complexity. In this work, we present an implementation of the nonlinear Fokker–Planck collision operator written in terms of Rosenbluth potentials in the Rosenbluth–MacDonald–Judd (RMJ) form. The Rosenbluth potentials may be obtained either by direct integration or by solving partial differential equations (PDEs) similar to Poisson's equation: we optimise for performance and scalability by using sparse matrices to solve the relevant PDEs. We represent the distribution function using a tensor-product continuous-Galerkin finite-element representation and we derive and describe the implementation of the weak form of the collision operator. We present tests demonstrating a successful implementation using an explicit time integrator and we comment on the speed and accuracy of the operator. Finally, we speculate on the potential for applications in the current and next generation of kinetic plasma models.
70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
Non-asymptotic analysis of ensemble Kalman updates: effective dimension and localization
Many modern algorithms for inverse problems and data assimilation rely on ensemble Kalman updates to blend prior predictions with observed data. Ensemble Kalman methods often perform well with a small ensemble size, which is essential in applications where generating each particle is costly. This paper develops a non-asymptotic analysis of ensemble Kalman updates, which rigorously explains why a small ensemble size suffices if the prior covariance has moderate effective dimension due to fast spectrum decay or approximate sparsity. Here, we present our theory in a unified framework, comparing everal implementations of ensemble Kalman updates that use perturbed observations, square root filtering and localization. As part of our analysis, we develop new dimension-free covariance estimation bounds for approximately sparse matrices that may be of independent interest.
Mathematics↗
Spray: Sparse Reductions of Arrays in OPEN MP
We present SPRAY, an open-source header-only C++ library for sparse reductions of arrays. SPRAY is meant for applications in which a large array is collaboratively updated by multiple threads using an associative and commutative operation such as +=. Especially when each thread accesses only parts of the array, SPRAY can perform significantly better than OPENMP's built-in reduction clause or atomic updates, while also using less memory than the former. SPRAY provides both an easy-to-use interface that can serve as a drop-in replacement for OPENMP reductions and a selection of reducer objects that accumulate the final result in different thread-safe ways. We demonstrate SPRAY through multiple test cases including the LULESH shock hydrodynamics code and a transpose-matrix-vector multiplication for sparse matrices stored in CSR format. SPRAY reductions outperform built-in OPENMP reductions consistently, in some cases improving run time and memory overhead by 20X, and even beating domain-specific approaches such as Intel MKI, by over 2X in some cases. Furthermore, SPRAY reductions have a minimal impact on the code base, requiring only a few lines of source code changes. Once in place, SPRAY reduction schemes can be switched easily, allowing performance portability and tuning opportunities by separating performance-critical implementation details from application code.
OpenMP↗