Parallel Implementation of the Density Matrix Renormalization Group Method Achieving a Quarter petaFLOPS Performance on a Single DGX-H100 GPU Node
Not Available
37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
SEARCH · Engineering Papers
Results for “Petaflops”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Direct simulation Monte Carlo on petaflop supercomputers and beyond
The gold-standard definition of the Direct Simulation Monte Carlo (DSMC) method is given in the 1994 book by Bird [Molecular Gas Dynamics and the Direct Simulation of Gas Flows (Clarendon Press, Oxford, UK, 1994)], which refined his pioneering earlier papers in which he first formulated the method. In the intervening 25 years, DSMC has become the method of choice for modeling rarefied gas dynamics in a variety of scenarios. The chief concern to applying DSMC to more dense or even continuum flows is its computational expense compared to continuum computational fluid dynamics methods. The dramatic (nearly billion-fold) increase in speed of the largest supercomputers over the last 30 years has thus been a key enabling factor in using DSMC to model a richer variety of flows, due to the method’s inherent parallelism. We have developed the open-source SPARTA DSMC code with the goal of running DSMC efficiently on the largest machines, both current and future. It is largely an implementation of Bird’s 1994 formulation. In this work, we describe algorithms used in SPARTA to enable DSMC to operate in parallel at the scale of many billions of particles or grid cells, or with billions of surface elements. We highlight a few examples of the kinds of fundamental physics questions and engineering applications that DSMC can address at these scales.
97 MATHEMATICS AND COMPUTING↗
Experiences Scaling a Production Arm Supercomputer to Petaflops and Beyond.
Abstract not provided.
Pedretti, Kevin↗
Argonne Leadership Computing Facility: 2021 Operational Assessment Report
This Operational Assessment Report describes how the Argonne Leadership Computing Facility (ALCF) met or exceeded every one of its goals for calendar year (CY) 2021 as an advanced scientific computing center. In CY 2021, the ALCF operated its production resource, Theta, an Intel-based Cray XC40 system (11.7-petaflops) augmented with 24 NVIDIA DGX A100-based nodes (3.9-petaflops) that supports diverse workloads, integrating data analytics with artificial intelligence (AI) training and learning in a single platform. In 2021, we began deploying Polaris, our newest 40- petaflops system, and augmented this powerful testbed system with an additional 28 nodes to support the integration of real-time experiments and HPC resources. We also deployed our two largest storage systems yet, named Grand and Eagle, that will bring new services to our users and will power data-driven research for years to come. Last year, Theta delivered a total of 20.8 million node-hours to 16 Innovative and Novel Computational Impact on Theory and Experiment (INCITE) projects and 7.2 million node-hours to ASCR Leadership Computing Challenge (ALCC) projects (32 awarded during the 2020–2021 ALCC year and 17 awarded during the 2021–2022 ALCC year), as well as substantial support to Director’s Discretionary (DD) projects (5.5 million node-hours). As Table ES.1 shows, Theta performed exceptionally well in terms of overall availability (95.1 percent), scheduled availability (99.4 percent), and utilization (98.1 percent; Table 2.1). As of the submission date of this document, ALCF’s user community has published 249 papers in high-quality, peer-reviewed journals and technical proceedings. At the 2021 International Conference for High Performance Computing, Networking, Storage and Analysis (SC’21), Argonne researchers won two HPCwire Readers’ Choice Awards and were part of a Gordon Bell Prize finalist team recognized for developing an AI-enabled, multi-resolution simulation framework for studying complex biomolecular machines. Their framework was used to observe the SARS-CoV-2 replication-transcription machinery in action, by directly integrating experimental data. ALCF also provided a comprehensive program of high-performance computing (HPC) support services to help our community make productive use of the facility’s diverse and growing collection of resources. We are now entering the exascale era, with exascale machines being planned for national laboratories across the country, including Aurora at Argonne National Laboratory (Argonne) in 2023. ALCF researchers have been leading and guiding numerous strategic activities that will push the boundaries of what’s possible in computational science and engineering and allow us to deliver science on day one.
97 MATHEMATICS AND COMPUTING↗
2022 Operational Assessment Report - Argonne Leadership Computing Facility
This Operational Assessment Report describes how the Argonne Leadership Computing Facility (ALCF) met or exceeded every one of its goals for the calendar year (CY) 2022. In CY 2022, ALCF operated Theta, an Intel-based Cray XC40 system (11.7 petaflops) augmented with 24 NVIDIA DGX A100-based nodes (3.9 petaflops), that supports diverse workloads, integrating data analytics with artificial intelligence (AI) training and learning in a single platform; and Polaris, a 44-petaflop AMD and NVIDIA-based HPE Apollo 6500 Gen10+ system that provides a powerful new platform to prepare applications and workloads for Aurora, Argonne National Laboratory’s (Argonne’s) upcoming Intel-Hewlett Packard Enterprise (HPE) exascale supercomputer.
97 MATHEMATICS AND COMPUTING↗
Language models for the prediction of SARS-CoV-2 inhibitors
The COVID-19 pandemic highlights the need for computational tools to automate and accelerate drug design for novel protein targets. We leverage deep learning language models to generate and score drug candidates based on predicted protein binding affinity. We pre-trained a deep learning language model (BERT) on ∼9.6 billion molecules and achieved peak performance of 603 petaflops in mixed precision. Our work reduces pre-training time from days to hours, compared to previous efforts with this architecture, while also increasing the dataset size by nearly an order of magnitude. For scoring, we fine-tuned the language model using an assembled set of thousands of protein targets with binding affinity data and searched for inhibitors of specific protein targets, SARS-CoV-2 Mpro and PLpro. We utilized a genetic algorithm approach for finding optimal candidates using the generation and scoring capabilities of the language model. Our generalizable models accelerate the identification of inhibitors for emerging therapeutic targets.
Blanchard, Andrew E.↗
Early experiences evaluating the HPE/Cray ecosystem for AMD GPUs
Summary The Oak Ridge Leadership Computing Facility (OLCF) has a long history of supporting and promoting GPU‐accelerated computing starting with the deployment of the Titan supercomputer in 2021 and continuing with the Summit supercomputer which has a theoretical peak performance of approximately 200 petaflops. Because the majority of Summit's computational power comes from its 27,972 GPUs, users must port their applications to one of the supported programming models in order to make efficient use of the system. To prepare the transition to Frontier, the OLCF's exascale supercomputer, users will need to adapt to an entirely new ecosystem which will include new hardware and software technologies. First, users will need to familiarize themselves with the AMD Radeon GPU architecture. Furthermore, users who have been previously relying on CUDA will need to transition to the Heterogeneous‐Computing Interface for Portability (HIP) or one of the other supported programming models (e.g., OpenMP, OpenACC). In this work, we describe our initial experiences and lessons learned in porting three applications or proxy apps currently running on Summit to the HPE/Cray ecosystem to leverage the compute power from AMD GPUs: minisweep, GenASiS, and Sparkler. Each one is representative of current production workloads utilized at the OLCF, different programming languages, and different programming models.
Melesse Vergara, Verónica G.↗
Scalable multiscale modeling of platelets with 100 million particles
Here, we developed the core components of the AI-aided multiple time stepping algorithm for multiscale modeling of cell dynamics. This algorithm was implemented and analyzed on two supercomputer architectures with an application of simulating the aggregation of 250 platelets, or 102 million particles. To scale on these computers with complex memory and network architectures with GPUs, we devised a biomechanics-informed task mapping scheme to optimize load imbalance, communications, and memory utilization. Our simulations, scaling well up to 192 nodes on a Summit-like supercomputer with a peak speed of 11 petaflops, achieved a rate of 423 μs/day which is 500 times faster than the conventional algorithm using static time step and this has enabled studies of record size blood clots at record spatial–temporal resolutions. Additionally, we discovered the sensitive dependence of the scalability and execution time on the methods of decomposition, CPU–GPU coupling, and task mapping.
97 MATHEMATICS AND COMPUTING↗
Study of interconnect errors, network congestion, and applications characteristics for throttle prediction on a large scale HPC system
Today’s High Performance Computing (HPC) systems contain thousand of nodes which work together to provide performance in the order of petaflops. The performance of these systems depends on various components like processors, memory, and interconnect. Among all, interconnect plays a major role as it glues together all the hardware components in an HPC system. A slow interconnect can impact a scientific application running on multiple processes severely as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks a study that explores different interconnect errors, congestion events and applications characteristics on a large-scale HPC system. In our previous work, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors, and congestion events. In this work, we first show how congestion events can impact application performance. We then investigate application characteristics interaction with interconnect errors and network congestion to predict applications encountering congestion with more than 90% accuracy.
97 MATHEMATICS AND COMPUTING↗
Toward Exascale: Overview of Large Eddy Simulations and Direct Numerical Simulations of Nuclear Reactor Flows with the Spectral Element Method in Nek5000
At the beginning of the last decade, Petascale supercomputers (i.e., computers capable of more than 1 petaFLOP) emerged. Now, at the dawn of exascale supercomputing, we provide a review of recent landmark simulations of portions of reactor components with turbulence-resolving techniques that this computational power has made possible. In fact, these simulations have provided invaluable insight into flow dynamics, which is difficult or often impossible to obtain with experiments alone. We focus on simulations performed with the spectral element method, as this method has emerged as a powerful tool to deliver massively parallel calculations at high fidelity by using large eddy simulation or direct numerical simulation. We also limit this paper to constant-property incompressible flow of a Newtonian fluid in the absence of other body or external forces, although the method is by no means limited to this class of flows. We briefly review the fundamentals of the method and the reasons it is compelling for the simulation of nuclear engineering flows. We review in detail a series of Petascale simulations, including the simulations of helical coil steam generators, fuel assemblies, and pebble beds. Even with Petascale computing, however, limitations for nuclear modeling and simulation tools remain. In particular, the size and scope of turbulence-resolving simulations are still limited by computing power and resolution requirements, which scale with the Reynolds number. In the final part of this paper, we discuss the future of the field, including recent advancements in emerging architectures such as GPUbased supercomputers, which are expected to power the next generation of high-performance computers.
computational fluid dynamics↗
ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability
Earth system predictability is challenged by the complexity of environmental dynamics and the multitude of variables involved. Current AI foundation models, although advanced by leveraging large and heterogeneous data, are often constrained by their size and data integration, limiting their effectiveness in addressing the full range of Earth system prediction challenges. To overcome these limitations, we introduce the Oak Ridge Base Foundation Model for Earth System Predictability (ORBIT), an advanced vision transformer model that scales up to 113 billion parameters using a novel hybrid tensor-data orthogonal parallelism technique. As the largest model of its kind, ORBIT surpasses the current climate AI foundation model size by a thousandfold. Performance scaling tests conducted on the Frontier supercomputer have demonstrated that ORBIT achieves 684 petaFLOPS to 1.6 exaFLOPS sustained throughput, with scaling efficiency maintained at 41% to 85% across 49,152 AMD GPUs. These breakthroughs establish new advances in AI-driven climate modeling and demonstrate promise to significantly improve the Earth system predictability.
Wang, Xiao↗
Characterization and identification of HPC applications at leadership computing facility
High Performance Computing (HPC) is an important method for scientific discovery via large-scale simulation, data analysis, or artificial intelligence. Leadership-class supercomputers are expensive, but essential to run large HPC applications. The Petascale era of supercomputers began in 2008, with the first machines achieving performance in excess of one petaflops, and with the advent of new supercomputers in 2021 (e.g., Aurora, Frontier), the Exascale era will soon begin. However, the high theoretical computing capability (i.e., peak FLOPS) of a machine is not the only meaningful target when designing a supercomputer, as the resources demand of applications varies. A deep understanding of the characterization of applications that run on a leadership supercomputer is one of the most important ways for planning its design, development and operation. In order to improve our understanding of HPC applications, user demands and resource usage characteristics, we perform correlative analysis of various logs for different subsystems of a leadership supercomputer. This analysis reveals surprising, sometimes counter-intuitive patterns, which, in some cases, conflicts with existing assumptions, and have important implications for future system designs as well as supercomputer operations. For example, our analysis shows that while the applications spend significant time on MPI, most applications spend very little time on file I/O. Combined analysis of hardware event logs and task failure logs show that the probability of a hardware FATAL event causing task failure is low. Combined analysis of control system logs and file I/O logs reveals that pure POSIX I/O is used more widely than higher level parallel I/O. Based on holistic insights of the application gained through combined and co-analysis of multiple logs from different perspectives and general intuition, we engineer features to "fingerprint" HPC applications. We use t-SNE (a machine learning technique for dimensionality reduction) to validate the explainability of our features and finally train machine learning models to identify HPC applications or group those with similar characteristic. To the best of our knowledge, this is the first work that combines logs on file I/O, computing, and inter-node communication for insightful analysis of HPC applications in production.
Liu, Zhengchun↗
Initial Industry Collaborations of the Center of Excellence
The NEAMS program aims to develop an integrated multi-physics simulation capability “pellet-to plant” for the design and analysis of future generations of nuclear power plants. In particular, the Reactor Product Line code suite's multi-resolution hierarchy is being designed to ultimately span the full range of length and time scales present in relevant reactor design and safety analyses, as well as scale from desktop to petaflop computing platforms. In particular the NEAMS program is supporting the development of novel thermal-hydraulic codes. The Center of Excellence for Thermal Fluids Application in Nuclear Energy, launched in 2018, has as its key goals to serve as a front door to industry. The Center of Excellence for Thermal Fluids Applications in Nuclear Energy has recently launched a program to start collaborative efforts between the laboratories and industry with the objective of stimulating cooperation and increasing adoption of T/H tools developed under NEAMS by the industry-at-large. In particular, two industry partners agreed to participate in pilot short-term collaborations aimed at demonstrating the value of NEAMS tools to their designs. These are BWXT and Kairos who proposed projects related to HTGR and FHR designs respectively. In this report we present the results of these collaborations.
22 GENERAL STUDIES OF NUCLEAR REACTORS↗
Demonstration of Fully Coupled Calculations in Helical Steam Generator: Toward Predictions of Fluid Elastic Instability
The NEAMS program aims to develop an integrated multi-physics simulation capability “pellet-to-plant” for the design and analysis of future generations of nuclear power plants. In particular, the Reactor Product Line code suite's multi-resolution hierarchy is being designed to ultimately span the full range of length and time scales present in relevant reactor design and safety analyses, as well as scale from desktop to petaflop computing platforms.
42 ENGINEERING↗
One-Way Coupled Flow-Induced Vibration Analysis for a Twisted-Tube Heat Exchanger
The NEAMS program aims to develop an integrated multi-physics simulation capability “pellet-to-plant” for the design and analysis of future generations of nuclear power plants. In particular, the Reactor Product Line code suite's multi-resolution hierarchy is being designed to ultimately span the full range of length and time scales present in relevant reactor design and safety analyses, as well as scale from desktop to petaflop computing platforms. Flow-induced vibration (FIV) is a widespread problem in energy systems because they rely on fluid movement for energy conversion. Vibrating structures may be damaged as fatigue or wear occurs. Given the importance of reliable components in the nuclear industry, flow-induced vibration has long been a major concern in safety and operation of nuclear reactors. Nuclear fuel rods and steam generators have been known to suffer from flow-induced vibration and related failures.
22 GENERAL STUDIES OF NUCLEAR REACTORS↗
Center of Excellence Collaboration Projects: Second Wave
The NEAMS program aims to develop an integrated multi-physics simulation capability “pellet-to-plant” for the design and analysis of future generations of nuclear power plants. In particular, the Reactor Product Line code suite's multi-resolution hierarchy is being designed to ultimately span the full range of length and time scales present in relevant reactor design and safety analyses, as well as scale from desktop to petaflop computing platforms. In particular the NEAMS program is supporting the development of novel thermal-hydraulic codes. The Center of Excellence for Thermal Fluids Application in Nuclear Energy, launched in 2018, has as its key goals to serve as a front door to industry. The Center of Excellence for Thermal Fluids Applications in Nuclear Energy has recently launched a program to start collaborative efforts between the laboratories and industry with the objective of stimulating cooperation and increasing adoption of T/H tools developed under NEAMS by the industry-at-large. In particular, two industry partners agreed to participate in a second wave of short-term collaborations aimed at demonstrating the value of NEAMS tools to their designs. These are Framatome and TerraPower who proposed projects related to LWR (Light Water Reactor) and MSFR (Molten Salt Fast Reactor) designs respectively. In this report we present the results of these two collaborations.
22 GENERAL STUDIES OF NUCLEAR REACTORS↗
The Multiphysics on Advanced Platforms Project
In 2015, the Lawrence Livermore National Laboratory started development of next-generation multiphysics simulation capabilities for the National Nuclear Security Administration under the Advanced Technologies Development and Mitigation (ATDM) element of the Advanced Simulation and Computing program in collaboration with the Exascale Computing Project (ECP). A key driver for this effort across the NNSA tri-lab was the emergence of advanced high performance computing (HPC) architectures based on heterogeneous compute capabilities, including GPU based systems, as part of the national drive toward exascale computing platforms at multiple Department of Energy (DOE) facilities. Developing a multiphysics code capable of meeting the various simulation needs of the NNSA as defined by the current generation of integrated codes (or ICs), initially developed as part of the Accelerated Strategic Computing Initiative (ASCI) program beginning in 1996, and able to scale to the current 100 petaflop class pre-exascale systems, as well the forthcoming exaflop class computers, is a daunting challenge. To accomplish this ambitious goal, LLNL has embraced two key themes: use of high-order numerical methods and a modular approach to code development. The LLNL next generation effort is organized under the Multi-Physics on Advanced Platforms Project (MAPP). A foundational component of MAPP is the Axom computer science (CS) toolkit which provides infrastructure for the development of modular, performance portable, multi-physics application codes. MARBL is a next-generation application code built on the Axom base to address the modeling needs of the high energy density physics (HEDP) community for simulating high-explosive, magnetic or laser driven experiments such as inertial confinement fusion (ICF), pulsed-power magneto-hydrodynamics (MHD), equation of state (EOS) and material strength studies as part of the NNSA’s stockpile stewardship program (SSP).
97 MATHEMATICS AND COMPUTING↗
IC reports (2021)
We requested HPC support to continue research on the seismic waves generated by impacts. We used the following codes: (1) the Hybrid Optimization Software Suite (HOSS), developed at LANL. HOSS is based on a combined Finite and Discrete Element Method (FDEM). New material models are developed for the sedimentary rocks. HOSS has been recently benchmarked to iSale and FLAG codes. SPECFEM3D is an open-source code developed since the last 90s. It won the Gordon Bell award for best performance in 2003, was finalist again in 2008 for a run at 0.16 petaflops on 149,784 cores on the ‘Jaguar’ Cray system at Oak Ridge National Laboratory. It also won the BULL Joseph Fourier supercomputing award in 2010.; SW4 is a 4th-order finite difference code developed at LLNL which is currently actively developed to handle complex 3D models and to be ported on future exascale platforms. We assessed our need to a total of 3.9M CPU-hrs for year 1 and and 3.1 M for year 2.
79 ASTRONOMY AND ASTROPHYSICS↗