Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “exascale computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,081 records · Page 60

High performance sparse multifrontal solvers on modern GPUs

Here, we have ported the numerical factorization and triangular solve phases of the sparse direct solver STRUMPACK to GPU. STRUMPACK implements sparse LU factorization using the multifrontal algorithm, which performs most of its operations in dense linear algebra operations on so-called frontal matrices of various sizes. Our GPU implementation off-loads these dense linear algebra operations, as well as the sparse scatter–gather operations between frontal matrices. For the larger frontal matrices, our GPU implementation relies on vendor libraries such as cuBLAS and cuSOLVER for NVIDIA GPUs and rocBLAS and rocSOLVER for AMD GPUs. For the smaller frontal matrices we developed custom CUDA and HIP kernels to reduce kernel launch overhead. Overall, high performance is achieved by identifying submatrix factorizations corresponding to sub-trees of the multifrontal assembly tree which fit entirely in GPU memory. The multi-GPU setting uses SLATE (Software for Linear Algebra Targeting Exascale) as a modern GPU-aware replacement for ScaLAPACK. On 4 nodes of SUMMIT the code runs ~10X faster when using all 24 V100 GPUs compared to when it only uses the 168 POWER9 cores. On 8 SUMMIT nodes, using 48 V100 GPUs, the sparse solver reaches over 50TFlop/s. Compared to SuperLU, on a single V100, for a set of 17 matrices our implementation is faster for all but one matrix, and is on average 5X (median 4X) faster

97 MATHEMATICS AND COMPUTING↗

Preparing NERSC users for Cori, a Cray XC40 system with Intel many integrated cores

The newest NERSC supercomputer Cori is a Cray XC40 system consisting of 2,388 Intel Xeon Haswell nodes and 9,688 Intel Xeon-Phi “Knights Landing” (KNL) nodes. Compared to the Xeon-based clusters NERSC users are familiar with, optimal performance on Cori requires consideration of KNL mode settings; process, thread, and memory affinity; fine-grain parallelization; vectorization; and use of the high-bandwidth MCDRAM memory. This paper describes our efforts preparing NERSC users for KNL through the NERSC Exascale Science Application Program, Web documentation, and user training. We discuss how we configured the Cori system for usability and productivity, addressing programming concerns, batch system configurations, and default KNL cluster and memory modes. Here, system usage data, job completion analysis, programming and running jobs issues, and a few successful user stories on KNL are presented.

97 MATHEMATICS AND COMPUTING↗

Towards Exascale Astrophysics of Mergers and Supernovae (TEAMS)

The TEAMS project brought together cutting-edge simulations, theoretical insights, and collaborative efforts to deepen our understanding of some of the universe’s most extreme phenomena—supernovae, neutron star mergers, and the powerful signals they emit. Using one of the largest suites of 3D supernova simulations ever conducted, researchers uncovered new insights into how massive stars explode, how those explosions vary by stellar mass, and what conditions lead to the birth of neutron stars or black holes. They also studied the radiation and gravitational wave signals emitted during these events, revealing how future observations can be used to uncover what happens deep inside collapsing stars. The team developed improved tools for modeling how light and neutrinos behave in such explosive environments, enabling more accurate predictions of what astronomers might observe. Work also explored how the chemical composition and geometry of kilonovae—the visible explosions that follow neutron star mergers—influence their signals and can reveal the origins of heavy elements like gold. These efforts not only advanced scientific knowledge, but also trained a new generation of researchers at the intersection of astrophysics, computational science, and nuclear theory.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

ExaChem/exachem

Open Source Exascale Quantum Chemistry Software

Panyala, Ajay [Pacific Northwest National Laborato↗

Performance Understanding and Analysis for Exascale Data Management Workflows (Collaboration)

The general approach of the MONA project is depicted in Figure 1. The figure shows performance monitoring applied to an I/O workflow associated with a scientific simulation, with online measurements captured from this workflow informing methods for workflow reconfiguration and dynamic adjustment. The goal is to maintain suitable levels of Quality of Service for workflow execution, by understanding the underlying causes of workflow performance and behavior. One outcome will be workflow performance models able to characterize realistic workflows. Another outcome will be performance ‘mini-apps’ implementing such workflow behavior. Out of scope for this project are advanced methods for online workflow control.

97 MATHEMATICS AND COMPUTING↗

Performance-portable Binary Neutron Star Mergers with AthenaK

We introduce an extension to the AthenaK code for general-relativistic magnetohydrodynamics (GRMHD) in dynamical spacetimes using a 3+1 conservative Eulerian formulation. Like the fixed-spacetime GRMHD solver, we use standard finite-volume methods to evolve the fluid and a constrained-transport scheme to preserve the divergence-free constraint for the magnetic field. We also utilize a first-order flux correction (FOFC) scheme to reduce the need for an artificial atmosphere and optionally enforce a maximum principle to improve robustness. We demonstrate the accuracy of AthenaK using a set of standard tests in flat and curved spacetimes. Using a SANE accretion disk around a Kerr black hole, we compare the new solver to the existing solver for stationary spacetimes using the so-called "HARM-like" formulation. We find that both formulations converge to similar results. We also include the first published binary neutron star (BNS) mergers performed on graphical processing units (GPUs). Thanks to the FOFC scheme, our BNS mergers maintain a relative error of $\mathcal{O}$(10 –11 ) or better in baryon mass conservation up to collapse. Finally, we perform scaling tests of AthenaK on OLCF Frontier, where we show excellent weak scaling of ≥80% efficiency up to 32,768 GPUs and 74% up to 65,536 GPUs for a GRMHD problem in dynamical spacetimes with six levels of mesh refinement. AthenaK achieves an order-of-magnitude speedup using GPUs compared to CPUs, demonstrating that it is suitable for performing numerical relativity problems on modern exascale resources.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Optimizing fusion PIC code performance at scale on Cori Phase 2

In this paper we present the results of optimizing the performance of the gyrokinetic full-f fusion PIC code XGC1 on the Cori Phase Two Knights Landing system. The code has undergone substantial development to enable the use of vector instructions in its most expensive kernels within the NERSC Exascale Science Applications Program. We study the single-node performance of the code on an absolute scale using the roofline methodology to guide optimization efforts. We have obtained 2x speedups in single node performance due to enabling vectorization and performing memory layout optimizations. On multiple nodes, the code is shown to scale well up to 4000 nodes, near half the size of the machine. We discuss some communication bottlenecks that were identified and resolved during the work.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

The Energy Exascale Earth System Model Simulations With High Vertical Resolution in the Lower Troposphere

Abstract General circulation models (GCMs) are typically run with coarse vertical resolution. For example, the Energy Exascale Earth System Model (E3SM) has a vertical resolution of about 200 m in the boundary layer, which is far too coarse to resolve sharp gradients often found in the thermodynamic fields capping subtropical marine stratocumulus. In this article, we present a series of multiyear atmosphere only simulations of E3SM version 1 where we progressively increase the vertical resolution in the lower troposphere to scales approaching those often used in large eddy simulation (LES). We report marginal impacts in regards to the simulation of boundary layer clouds when vertical resolution is moderately increased, yet find significant positive impacts when the vertical resolution approaches that typically used in LES (∼10 m). In these experiments, there is a marked change in the simulated turbulence and thermodynamics which leads to more abundant marine stratocumulus. However, these simulations are burdened with excessive computational cost. They are also subject to degradations in overall climate metrics due to time step sensitivities and because some processes and parameterizations are sensitive to changes in the vertical resolution.

54 ENVIRONMENTAL SCIENCES↗

exadigitUE5

This project provides the AR/VR interface to ORNL's exascale digital twin. The main functionality is implemented using Unreal Engine 5.1 for Desktop or Microsoft Hololens2 based visualization and interation with the system. The digital twin provides data ingestion from telemetry, as well as triggering and interacting with simulations developed for the wider ExaDigiT project at ORNL, as well as for the LUMI system at CSC and other CrayEX Supercomputers. For the overarching project, see ExaDigiT at https://exadigit.github.io, with the code repositories at https://code.ornl.gov/exadigit.

Maiterth, Matthias [Oak Ridge National Laboratory ↗

Energy-conserving physics for nonhydrostatic dynamics in mass coordinate models

Abstract. Motivated by reducing errors in the energy budget related to enthalpy fluxes within the Energy Exascale Earth System Model (E3SM), we study several physics–dynamics coupling approaches. Using idealized physics, a moist rising bubble test case, and the E3SM's nonhydrostatic dynamical core, we consider unapproximated and approximated thermodynamics applied at constant pressure or constant volume. With the standard dynamics and physics time-split implementation, we describe how the constant-pressure and constant-volume approaches use different mechanisms to transform physics tendencies into dynamical motion and show that only the constant-volume approach is consistent with the underlying equations. Using time step convergence studies, we show that the two approaches both converge but to slightly different solutions. We reproduce the large inconsistencies between the energy flux internal to the model and the energy flux of precipitation when using approximate thermodynamics, which can only be removed by considering variable latent heats, both when computing the latent heating from phase change and when applying this heating to update the temperature. Finally, we show that in the nonhydrostatic case, for physics applied at constant pressure, the general relation that enthalpy is locally conserved no longer holds. In this case, the conserved quantity is enthalpy plus an additional term proportional to the difference between hydrostatic pressure and full pressure.

58 GEOSCIENCES↗

Enabling Exascale Hardware and Software Design through Scalable System Virtualization (Final Report)

In this grant, we enhanced the Palacios virtual machine monitor to increase its scalability and suitability for addressing exascale system software design issues. This included a wide range of research on core Palacios features, large-scale system emulation, fault injection, perfomrance monitoring, and VMM extensibility. This research resulted in large number of high-impact publications in well-known venues, the support of a number of students, and the graduation of two Ph.D. students and one M.S. student. In addition, our enhanced version of the Palacios virtual machine monitor has been adopted as a core element of the Hobbes operating system under active DOE-funded research and development.

97 MATHEMATICS AND COMPUTING↗

MCNP6 Monte Carlo Code Optimization [Slides]

The MCNP6.1 Monte Carlo code, released in 2013, offers many features not available in previous versions, but runs 20-30% slower for most problems and 2-5x slower for some problems. The MCNP 2020 initiative was established in mid-2013 to address code performance and other issues, with 3 basic components: (1) Improve MCNP performance, with the specific goal of achieving a 2x performance gain within 2 years; (2) Upgrade the core MCNP software and algorithms in an evolutionary fashion, with the goal of achieving more agile and sustainable code; (3) Prepare for future exascale systems that will include massive parallelism and limited memory per core, with the goal of providing flexible and adaptable coding for possible exascale systems. This talk reviews the initial performance improvements to MCNP6.1 that have been incorporated into the 2014 update, MCNP6.1.1. The performance improvements to date have included both classic code optimizations and algorithm improvements. Testing on a variety of problems has demonstrated that the performance improvements were effective, yielding speedups by factors of 1.2x - 4x compared to MCNP6.1, depending on the type of problem. For criticality problems, MCNP6.1.1 runs 1.5x - 1.7x faster than MCNP6.1. Much more work is planned for improving MCNP6 performance, structure, and algorithms.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Evaluating Trade-offs in Potential Exascale Interconnect Topologies

This study looks at the high-level trade-offs for interconnect architectures that are likely to be available in HPC machines in the 2021-2022 timeframe. The study uses multiple simulation models to look at the effects of varying machine size, injection bandwidth and topology for a selected set of benchmark communication patterns. The studies show the sensitivity of various applications to these network parameters. This study can help inform decisions on node size versus number of nodes, which topology is most efficient for our applications, how much injection bandwidth to provision, etc. These simulations only look at interconnects with maximally configured global bandwidth, and studies of networks configured with less will be the subject of future milestones.

97 MATHEMATICS AND COMPUTING↗

Hydraulic Conductivity Measurements, Utqiagvik (Barrow), Alaska, 2014

Six individual ice cores were collected from the Barrow Environmental Observatory in Barrow, Alaska, in May of 2013 as part of the Next Generation Ecosystem Experiment (NGEE). Each core was drilled at a different location to varying depths. After drilling, the cores were stored in coolers packed with dry ice and flown to Lawrence Berkeley National Laboratory (LBNL) in Berkeley, CA. 3-dimensional images of the cores were constructed using medical X-ray computed tomography (CT) scanner at 120kV. Hydraulic conductivity samples were extracted from these cores at LBNL Richmond Field Station in Richmond, CA, in February 2014 by cutting 5 to 8 inch segments using a chop saw. Samples were packed individually and stored at -20C freezing temperatures to minimize any changes in structure or loss of ice content prior to analysis. Hydraulic conductivity was determined through falling head tests using a permeameter [ELE International, Model #: K-770B] (Appendix A). Samples were placed in a latex membrane via a membrane stretcher while frozen. Use of a membrane stretcher made the membranes easier to secure and minimized contact with the sample. A clear polycarbonate sleeve, fabricated with a stainless steel ring at the bottom to keep the sleeve from floating, was placed around the sample inside the permeameter to minimize deformation during analysis. The permeameter was filled with water and 1.0 PSI of air was applied for confining pressure during sample defrost. Outflow valves were left open to allow for incremental thawing and samples were left to thaw for approximately 12 hours. After approximately 12 hours of thaw, initial falling head tests were performed. When the flow was significantly too fast or too slow, the analysis was stopped and the burette size was adjusted accordingly (i.e. a larger diameter burette was used for flows that were faster than desired or a smaller diameter burette was used for flows that were slower than desired). Two to four measurements were collected on each sample and collection stopped when the applied head load exceeded 25% change from the original load. Analyses were performed between 2 to 3 times for each sample. The final hydraulic conductivity calculations were computed using methodology of Das et al., 1985.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a 15-year research effort (2012-2027) to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

Exploring dynamic load imbalance solutions with the CoMD proxy application

Proxy applications are developed to simplify studying parallel performance of scientific simulations and to test potential solutions for performance problems. However, proxy applications are typically too simple to allow work migration or to represent the load imbalance of their parent applications. To study the ability of load balancing solutions to balance work effectively, we enable work migration in one of the Exascale Co-design Center for Materials in Extreme Environments (ExMatEx) [1] applications, CoMD. We design a methodology to parameterize three key aspects necessary for studying load imbalance correction: (1) the granularity with which work can be migrated; (2) the initial load imbalance; (3) the dynamic load imbalance (how quickly the load changes over time). Finally, we present a study of the impact of flexibility in work migration in CoMD on load balance and the associated rebalancing costs for a wide range of initial and dynamic load imbalance scenarios.

97 MATHEMATICS AND COMPUTING↗

Validation of time-dependent shift using the pulsed sphere benchmarks

The detailed behavior of neutrons in a rapidly changing time-dependent physical system is a challenging computational physics problem, particularly when using Monte Carlo methods on heterogeneous high-performance computing architectures. A small number of algorithms and code implementations have been shown to be performant for time-independent (fixed source and k-eigenvalue) Monte Carlo, and there are existing simulation tools that successfully solve the time-dependent Monte Carlo problem on smaller computing platforms. To bridge this gap, a time-dependent version of ORNL’s Shift code has been recently developed. Shift’s history-based algorithm on CPUs, and its event-based algorithm on GPUs, have both been observed to scale well to very large numbers of processors, which motivated the extension of this code to solve time-dependent problems. The validation of this new capability requires a comparison with time-dependent neutron experiments. Lawrence Livermore National Laboratory’s (LLNL) pulsed sphere benchmark experiments were simulated in Shift to validate both the time-independent as well as new time-dependent features recently incorporated into Shift. A suite of pulsed-sphere models was simulated using Shift and compared to the available experimental data and simulations with MCNP. Overall results indicate that Shift accurately simulates the pulsed sphere benchmarks, and that the new time-dependent modifications of Shift are working as intended. Validated exascale neutron transport codes are essential for a wide variety of future multiphysics applications.

Palmer, Camille J.↗

Modeling the Spatio-Temporal Variability in Subsurface Thermal Regimes Across a Low-Relief Polygonal Tundra Landscape: Modeling Archive

Vast carbon stocks stored in permafrost soils of Arctic tundra are under risk of release to atmosphere under warming climate. Ice--wedge polygons in the low-gradient polygonal tundra create a complex mosaic of microtopographic features. The microtopography plays a critical role in regulating the fine scale variability in thermal and hydrological regimes in the polygonal tundra landscape underlain by continuous permafrost. Modeling of thermal regimes of this sensitive ecosystem is essential for understanding the landscape behaviour under current as well as changing climate. We present here an end-to-end effort for high resolution numerical modeling of thermal hydrology at real-world field sites, utilizing the best available data to characterize and parameterize the models. We develop approaches to model the thermal hydrology of polygonal tundra and apply them at four study sites at Barrow, Alaska spanning across low to transitional to high-centered polygon and representative of broad polygonal tundra landscape. A multi--phase subsurface thermal hydrology model (PFLOTRAN) was developed and applied to study the thermal regimes at four sites. Using high resolution LiDAR DEM, microtopographic features of the landscape were characterized and represented in the high resolution model mesh. Best available soil data from field observations and literature was utilized to represent the complex heterogeneous subsurface in the numerical model. This data collection provides the complete set of input files, forcing data sets and computational meshes for simulations using PFLOTRAN for four sites at Barrow Environmental Observatory. It also documents the complete computational workflow for this modeling study to allow verification, reproducibility and follow up studies. This dataset includes two zipped files and one .pdf file.The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research.The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska.Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).

54 ENVIRONMENTAL SCIENCES↗

Program Correctness, Verification and Testing for Exascale (Corvette)

The goal of this project is to provide tools to assess the correctness of parallel programs written using hybrid parallelism. There is a dire lack of both theoretical and engineering know-how in the area of finding bugs in hybrid or large scale parallel programs, which our research aims to change. In the project we have demonstrated novel approaches in several areas: 1. Low overhead automated and precise detection of concurrency bugs at scale. 2. Using low overhead bug detection tools to guide speculative program transformations for performance. 3. Techniques to reduce the concurrency required to reproduce a bug using partial program restart/replay. 4. Techniques to provide reproducible execution of floating point programs. 5. Techniques for tuning the floating point precision used in codes.

97 MATHEMATICS AND COMPUTING↗