Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “tensor cores”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 records
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Accelerating GNNs on GPU Sparse Tensor Cores through N:M Sparsity-Oriented Graph Reordering

Recent advancements in GPU hardware support have introduced the capability to leverage N:M sparse patterns for substantial performance gains. Graphs in Graph Neural Networks (GNNs) are typically sparse, but the sparsity is often irregular, not conforming to such sparse patterns. In this paper, we propose a novel graph reordering algorithm, the first of its kind, to reshape irregular graph data into the N:M structured sparse pattern at the tile level, allowing linear-algebra-based graph operations in GNNs to benefit from the N:M sparse hardware. The optimization is lossless, maintaining the accuracy of GNN. It can remove 98-100\% violations of the N:M sparse patterns at the vector level, and increase the proportion of conforming graphs in SuiteSparse collection from 5-9\% to 88.7-93.5\%. On A100 GPUs, the optimization accelerates Sparse Matrix Matrix (SpMM) by up to 43X (2.3X -- 7.5X on average) and speeds up the key graph operations in GNNs on real graphs by as much as 8.6X (3.5X on average).

artificial intelligence, graph neural networks↗
DOE OSTI2022

Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numeric Behaviors

Tensor Cores have been an important unit to accelerate Fused Matrix Multiplication Accumulation (MMA) in all NVIDIA GPUs since Volta Architecture. To program Tensor Cores, users have to use either legacy wmma APIs or current mma APIs. Legacy wmma APIs are more easy-to-use but can only exploit limited features and power of Tensor Cores. Specifically, wmma APIs support fewer operand shapes and can not leverage the new sparse matrix multiplication feature of the newest Ampere Tensor Cores. However, the performance of current programming interface has not been well explored. Furthermore, the computation numeric behaviors of low-precision floating points (TF32, BF16, and FP16) supported by the newest Ampere Tensor Cores are also mysterious. In this paper, we explore the throughput and latency of current programming APIs. Further, we intuitively study the numeric behaviors of Tensor Cores MMA and profile the intermediate operations including multiplication, addition of inner product, and accumulation. All codes used in this work can be found in https://github.com/sunlex0717/DissectingTensorCores.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Mixed-Precision S/DGEMM Using the TF32 and TF64 Frameworks on Low-Precision AI Tensor Cores

Using NVIDIA graphics processing units (GPUs) equipped with Tensor Cores has enabled the significant acceleration of general matrix multiplication (GEMM) for applications in machine learning (ML) and artificial intelligence (AI) and in high-performance computing (HPC) generally. The use of such power-efficient, specialized accelerators can provide a performance increase between 8 × and 20 ×, albeit with a loss in precision. However, a high level of precision is required in many large scientific and HPC applications, and computing in single or double precision is still necessary for many of these applications to maintain accuracy. Fortunately, mixed-precision methods can be employed to maintain a higher level of numerical precision while also taking advantage of the performance increases from computing with lower-precision AI cores. With this in mind, we extend the state of the art by using NVIDIA’s new TF32 framework. This new framework not only burdens some constraints of the previous frameworks, such as costly 32 16-bit castings but also provides an equivalent precision and performance by using a much simpler approach. We also propose a new framework called TF64 that attempts double-precision arithmetic with low-precision Tensor Cores. Although this framework does not exist yet, we validated the correctness of this idea and achieved an equivalent of 64-bit precision on 32-bit hardware.

Valero Lara, Pedro↗
DOE OSTI2024

Kernel fusion in atomistic spin dynamics simulations on Nvidia GPUs using tensor core

In atomistic spin dynamics simulations, the time cost of constructing the space- and time-displaced pair correlation function in real space increases quadratically as the number of spins N, leading to significant computational effort. The GEMM subroutine can be adopted to accelerate the calculation of the dynamical spin-spin correlation function, but the computational cost of simulating large spin systems (>40000 spins) on CPUs remains expensive. In this work, we perform the simulation on the graphics processing unit (GPU), a hardware solution widely used as an accelerator for scientific computing and deep learning. Here we show that GPUs can accelerate the simulation up to 25-fold compared to multi-core CPUs when using the GEMM subroutine on both. To hide memory latency, we fuse the element-wise operation into the GEMM kernel using CUTLASS that can improve the performance by 26% ~ 33% compared to implementation based on cuBLAS. Furthermore, we perform the on-the-fly calculation in the epilogue of the GEMM subroutine to avoid saving intermediate results on global memory, which makes the large-scale atomistic spin dynamics simulation feasible and affordable.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Distributed out-of-memory NMF on CPU/GPU architectures

We propose an efficient distributed out-of-memory implementation of the non-negative matrix factorization (NMF) algorithm for heterogeneous high-performance-computing systems. The proposed implementation is based on prior work on NMFk, which can perform automatic model selection and extract latent variables and patterns from data. In this work, we extend NMFk by adding support for dense and sparse matrix operation on multi-node, multi-GPU systems. The resulting algorithm is optimized for out-of-memory problems where the memory required to factorize a given matrix is greater than the available GPU memory. Memory complexity is reduced by batching/tiling strategies, and sparse and dense matrix operations are significantly accelerated with GPU cores (or tensor cores when available). Input/output latency associated with batch copies between host and device is hidden using CUDA streams to overlap data transfers and compute asynchronously, and latency associated with collective communications (both intra-node and inter-node) is reduced using optimized NVIDIA Collective Communication Library (NCCL) based communicators. Benchmark results show significant improvement, from 32X to 76x speedup, with the new implementation using GPUs over the CPU-based NMFk. Good weak scaling was demonstrated on up to 4096 multi-GPU cluster nodes with approximately 25,000 GPUs when decomposing a dense 340 Terabyte-size matrix and an 11 Exabyte-size sparse matrix of density 10 -6 .

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Efficient Mixed-Precision Matrix Factorization of the Inverse Overlap Matrix in Electronic Structure Calculations with AI-Hardware and GPUs

In recent years, a new kind of accelerated hardware has gained popularity in the artificial intelligence (AI) community which enables extremely high-performance tensor contractions in reduced precision for deep neural network calculations. In this article, we exploit Nvidia Tensor cores, a prototypical example of such AI-hardware, to develop a mixed precision approach for computing a dense matrix factorization of the inverse overlap matrix in electronic structure theory, S –1 . This factorization of S –1 , written as ZZT = S –1 , is used to transform the general matrix eigenvalue problem into a standard matrix eigenvalue problem. Here we present a mixed precision iterative refinement algorithm where Z is given recursively using matrix–matrix multiplications and can be computed with high performance on Tensor cores. To understand the performance and accuracy of Tensor cores, comparisons are made to GPU-only implementations in single and double precision. Additionally, we propose a nonparametric stopping criteria which is robust in the face of lower precision floating point operations. The algorithm is particularly useful when we have a good initial guess to Z, for example, from previous time steps in quantum-mechanical molecular dynamics simulations or from a previous iteration in a geometry optimization.

36 MATERIALS SCIENCE↗
DOE OSTI2025

Tree tensor network hierarchical equations of motion based on time-dependent variational principle for efficient open quantum dynamics in structured thermal environments

In this work, we introduce an efficient method, TTN-HEOM, for exactly calculating the open quantum dynamics for driven quantum systems interacting with highly structured bosonic baths by combining the tree tensor network (TTN) decomposition scheme with the bexcitonic generalization of the numerically exact hierarchical equations of motion (HEOM). The method yields a series of quantum master equations for all core tensors in the TTN that efficiently and accurately capture the open quantum dynamics for non-Markovian environments to all orders in the system–bath interaction. These master equations are constructed based on the time-dependent Dirac–Frenkel variational principle, which isolates the optimal dynamics for the core tensors given the TTN ansatz. The dynamics converges to the HEOM when increasing the rank of the core tensors, a limit in which the TTN ansatz becomes exact. We introduce TENSO, tensor equations for non-Markovian structured open systems, as a general-purpose Python code to propagate the TTN-HEOM dynamics. We implement three general propagators for the coupled master equations: two fixed-rank methods that require a constant memory footprint during the dynamics and one adaptive-rank method with a variable memory footprint controlled by the target level of computational error. We exemplify the utility of these methods by simulating a two-level system coupled to a structured bath containing one Drude–Lorentz component and eight Brownian oscillators, which is beyond what can presently be computed using the standard HEOM. Our results show that the TTN-HEOM is capable of simulating both dephasing and relaxation dynamics of driven quantum systems interacting with structured baths, even those of chemical complexity, with an affordable computational cost.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2024

FTTN: Feature-Targeted Testing for Numerical Properties of NVIDIA & AMD Matrix Accelerators

While NVIDIA has been the dominant provider of GPUs for HPC and ML, now AMD has several offerings of GPUs. This encourages programmers to try out AMD GPUs for new codes and also port existing codes over. Unfortunately, without understanding the floating-point differences between these GPU types, software development or porting can introduce bugs—and currently such an understanding is lacking. The magnitude of this open question becomes clear if one imagines the the number of floating-point precision choices (FP16, FP32, etc.), floating-point formats (standard floats, brain-float, etc.), and execution units available (elementary units, matrix/tensor cores, etc.) Questions such as rounding modes and subnormal support are also important. Most of these answers are unknown today or are hard to access. We provide the first testing-guided approach that answers a significant number of these questions. We also devise tests to reveal internal information (e.g., extra bits kept) to make sure that our findings are reliable. Many of our tests employ systematically generated random-programs, others apply fast-math flags and some involve fused multiplyadd. Especially for tensor/matrix cores, the tests have nontrivial logic that we present Our testing approach is reusable for the plethora of GPUs yet to be introduced. Our findings include up to 7 ulps of difference between NVIDIA and AMD for sin and cos at FP32 precision and 3 ulp at FP64. In our study of matrix cores (NVIDIA) and tensor cores (AMD), we have extensively characterized rounding modes (truncation versus round-to-nearest), the number of extra internal bits kept (whether 3 bits are kept or not), subnormal support for inputs and outputs across four different floating-point formats and across NVIDIA A100 and AMD MI250X GPUs. We believe that this wealth of data becoming available for the first time may help avoid significant porting bugs when migrating code across these platforms.

Li, Xinyi↗
DOE OSTI2024

Towards High-Performance AI4NP Applications on Modern GPU Platforms

The evolution of modern heterogeneous accelerators, such as GPUs, has significantly advanced the landscape of artificial intelligence (AI). There is a notable surge to adopt AI within the nuclear physics domain (AI4NP). While most AI4NP studies focus on feasibility analysis, our attention is directed towards evaluating their performance on contemporary GPUs that integrate tensor cores. We first benchmark the throughput of hyperparameterized multi-layer perceptron (MLP) models. We then examine the performance of an AI4NP application: Hydra. We assess the performance gain and accuracy loss caused by the tensor cores for low-precision floating-point operations. Our experiments encompass the PyTorch and TensorFlow Keras frameworks on NVIDIA’s T4 and A100 GPUs. We explore the behavior of different GPU hardware platforms and AI software tools. This study can be a valuable resource for guiding the performance optimization of larger-scale deployments of AI4NP applications.

Mei, Xinxin↗
DOE OSTI2022

Towards Precision-Aware Fault Tolerance Approaches for Mixed-Precision Applications

Graphics Processing Units (GPUs), the dominantly adopted accelerators in HPC systems, are susceptible to transient hardware fault. New generation of GPUs feature mixed-precision architectures such as NVIDIA Tensor Cores to accelerate matrix multiplications. While being widely adapted, how would they behave under transient hardware faults remain unclear. In this study, we conduct a large-scale fault injection experiments on GEMM kernels implemented with different floating-point data types on the V100 and A100 Tensor Cores, and show distinct error resilience characteristics for the GEMMS with different formats. In the future, we plan to explore this space by building precision-aware floating-point fault tolerance techniques for applications such as DNNs that exercise low-precision computations.

Fang, Bo↗
DOE OSTI2024

Atomic Structure and Dynamics of Unusual and Wide-Gap Phase-Change Chalcogenides: A GeTe 2 Case

Brain-inspired computing, reconfigurable optical metamaterials, photonic tensor cores, and many other advanced applications require next-generation phase-change materials (PCMs) with better energy efficiency and a wider thermal and spectral range for reliable operations. Germanium ditelluride (GeTe 2 ), with higher thermal stability and a larger bandgap compared to current benchmark PCMs, appears promising for THz metasurfaces and the controlled crystallization of atomically thin 2D materials. Using high-energy X-Ray diffraction supported by first-principles simulation, the atomic structure in semiconducting pulsed laser deposition films and metallic high-temperature liquids is investigated. The results suggest that the structural and chemical metastability of GeTe 2 , leading to disproportionation into GeTe and Te, is related to high internal pressure during a semiconductor–metal transition, presumably occurring in the supercooled melt. Similar phenomena are expected for canonical GeS 2 and GeSe 2 under high temperatures and pressures.

74 ATOMIC AND MOLECULAR PHYSICS↗
DOE OSTI2026

Fragile-to-strong transition in liquid As 2 S 3 under pressure: The effect of melt metallization

The well-known classification of glass-forming melts into fragile and strong liquids has several notable exceptions, including water, silica, and certain phase-change materials (PCMs). These exceptional fluid systems exhibit a fragile-to-strong transition (FST) upon cooling: a transformation from a high-temperature liquid with fast atomic dynamics, low viscosity, and low flow activation energy, to a viscous supercooled melt with high energy barriers near the glass transition temperature T g . This behavior is critically important for non-volatile memories, photonic tensor cores, reconfigurable metamaterials, and other devices, that use PCMs, enabling nanosecond-scale crystallization in the fragile regime and long data retention in the strong regime near or below T g . A significant structural transformation is expected between these two viscosity regimes, along with a semiconductor-metal (SC-M) transition upon heating, driven by high internal pressure and associated density increase. By applying high external pressure to the canonical low-conducting chalcogenide melt As 2 S 3 , we observed both the FST and the SC-M transition, occurring simultaneously within the same domain of the P, T−phase space. These findings suggest that the FST is not limited to a few exceptional liquids but is a common phenomenon, at least in systems that exhibit melt metallization within specific regions of their P, T−phase diagrams.

first-principles molecular dynamics↗
DOE OSTI2025

Atomic Structure, Dynamics, Changes in Chemical Bonding and Semiconductor-Metal Transition in Sb 2 Se 3 : A Remarkable Material for Quantum Networks and Energy Applications

Antimony sesquiselenide has become an outstanding functional material for photovoltaics, energy storage and transformation, memory and photonic applications. Sb 2 Se 3 is one of the most successful emerging solar light absorbers and has also been identified as a highly promising ultralow-loss phase-change material (PCM) for next-generation coherent nanophotonic processors, photonic tensor cores, quantum and neuromorphic networks. Unlike benchmark telluride PCMs, Sb 2 Se 3 features a quasi-one-dimensional (1D) crystalline structure consisting of (Sb 4 Se 6 ) ∞ ribbons, lacks the typical PCM chemical bonding, and undergoes an extended semiconductor-metal transition above the melting point. Consequently, the origin of high optical contrast between crystalline (SET) and amorphous (RESET) logic states remains elusive and presents a significant challenge. Using high-energy X-ray diffraction and Raman spectroscopy over a wide temperature range, supported by first-principles simulations and complemented by thermal, optical and electrical measurements, as well as by 121 Sb-Mossbauer spectroscopy, the quasi-1D network of orthorhombic antimony sesquiselenide was found to undergo significant evolution in amorphous and supercooled Sb 2 Se 3 , leading to lower coordination, shorter interatomic distances and a higher p-electron density on antimony, indicating changes in chemical bonding. The observed novel Sb 2 Se 3 nanocrystalline polymorph, characterized by trigonal antimony coordination and more isolated Sb-Se ribbons, could help reduce multiple trapping defect states in the bandgap, which are typical of orthorhombic Sb 2 Se 3 , thereby enhancing the power-conversion efficiency of photovoltaic devices. Semimetallic and metallic liquid Sb 2 Se 3 exhibit a gradual transformation into a denser 2D and/or 3D network with higher antimony coordination. Localized electron states in the pseudogap are becoming extended, leading to an increase in electronic conductivity σ following the relationship σ ∝ N(E F ) 2 . Liquid Sb 2 Se 3 also appears to be strongly fragile, with a nonmonotonic change in viscosity and higher atomic mobility in the metallic liquid. Furthermore, these results explain extraordinary functionalities of Sb 2 Se 3 for photonic and energy applications.

antimony↗
DOE OSTI2024

Understanding Mixed Precision GEMM with MPGemmFI: Insights into Fault Resilience

Emerging deep learning workloads urgently need fast general matrix multiplication (GEMM). Thus, one of the critical features of machine-learning-specific accelerators such as NVIDIA Tensor Cores, AMD Matrix Cores, and Google TPUs is the support of mixed-precision enabled GEMM. For DNN models, lower-precision FP data formats and computation offer acceptable correctness but significant performance, area, and memory footprint improvement. While promising, the mixed-precision computation on error resilience remains unexplored. To this end, we develop a fault injection framework that systematically injects fault into the mixed-precision computation results. We investigate how the faults affect the accuracy of machine learning applications. Based on the characteristics of error resilience, we offer lightweight error detection and correction solutions that significantly improve the overall model accuracy by 75% if the models experience hardware faults. The solutions can be efficiently integrated into the accelerator's pipelines.

Fang, Bo↗
DOE OSTI2024

Bridging the Gap Between LLMs and LNS with Dynamic Data Format and Architecture Codesign

Deep neural networks (DNNs) have achieved tremendous success in the past few years. However, their training and inference demand exceptional computational and memory resources. Quantization has been shown as an effective approach to mitigate the cost, with the mainstream data types reduced from FP32 to FP16/BF16 and recently FP8 in the latest NVIDIA H100 GPUs. With increasingly aggressive quantization, however, the conventional floating-point formats suffer from limited precision in representing numbers around zero. Recently, NVIDIA demonstrated the potential of using a Logarithmic Number System (LNS) for the next generation of tensor cores. While LNS mitigates the hurdles in representing small numbers, in this work we observed a mismatch between LNS and the emerging Large Language Models (LLM), where LLM exhibits significant outliers when directly adopting the LNS format. In this paper, we present a data-format/architecture codesign to bright this gap. On the format side, we propose a dynamic LNS format to flexibly represent outliers at a higher precision, by exploiting asymmetry in the LNS representation and identifying outliers through a per-vector basis. On the architecture side, for demonstration, we realize the dynamic LNS format in a systolic array, which can handle the irregularity of the outliers at runtime. We implement our approach on an Alveo U280 FPGA as a prototype. Experimental results show that our design can effectively handle the outliers and resolve the mismatch between LNS and LLM, contributing to an accuracy improvement of 15.4% and 16% over the floating-point and the original LNS baselines, using four state-of-the-art LLM models. Our observation and design lay a solid foundation for the large-scale adoption of the LNS format in the next-generation deep learning hardware.

Haghi, Pouya↗
DOE OSTI2024

Real-time High-resolution X-Ray Computed Tomography

Computed Tomography (CT) serves as a key imaging technology that relies on computationally intensive filtering and back-projection algorithms for 3D image reconstruction. While conventional high-resolution image reconstruction (> 2K3) solutions provide quick results, they typically treat reconstruction as an offline workload to be performed remotely on large-scale HPC systems. The growing demand for post-construction AI-driven analytics and the need for real-time adjustments call for high-resolution reconstruction solutions that are feasible on local computing resources, i.e. a multi-GPU server at most. In this paper, we propose a novel approach that utilizes Tensor Cores to optimize image reconstruction without sacrificing precision. We also introduce a framework designed to enable real-time execution of end-to-end distributed image reconstruction in a multi-GPU environment. Evaluations conducted on a single Nvidia A100 and H100 GPU show performance improvements of 1.91 × and 2.15 × compared to highly optimized production libraries. Furthermore, our framework, when deployed on 8-card Nvidia A100 GPU system, demonstrates the ability to reconstruct real-world datasets into 20483 volumes (32 GB) in slightly more than one minute and 40963 volumes (256 GB) in 7 minutes.

Wu, Du↗