Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “EXASCALE”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,405 records · Page 78

Modes of Variability in E3SM and CESM Large Ensembles

An adequate characterization of internal modes of climate variability (MoV) is a prerequisite for both accurate seasonal predictions and climate change detection and attribution. Assessing the fidelity of climate models in simulating MoV is therefore essential; however, doing so is complicated by the large intrinsic variations in MoV and the limited span of the observational record. Large ensembles (LEs) provide a unique opportunity to assess model fidelity in simulating MoV and quantify intermodel contrasts. Here, in this work, these goals are pursued in four recently produced LEs: the Energy Exascale Earth System Model (E3SM) versions 1 and 2 LEs, and the Community Earth System Model (CESM) versions 1 and 2 LEs. In general, the representation of global coupled modes is found to improve across successive E3SM and CESM versions in conjunction with the fidelity of the base state climate while the patterns of extratropical modes are well simulated across the ensembles. Various persistent shortcomings for all MoV are however identified and discussed. The results both demonstrate the successes of these recent model versions and suggest the potential for continued improvement in the representation of MoV with advances in model physics.

54 ENVIRONMENTAL SCIENCES↗

Effects of Surface Turbulence Flux Parameterizations on the MJO: The Role of Ocean Surface Waves

This study investigates the sensitivity of the Madden–Julian oscillation (MJO) to changes to the bulk flux parameterization and the role of ocean surface waves in air–sea coupling using a fully coupled ocean–atmosphere–wave model. The atmospheric and ocean model components of the Energy Exascale Earth System Model (E3SM) are coupled to a spectral wave model, WAVEWATCH III (WW3). Two experiments with wind speed–dependent bulk algorithms (NCAR and COARE3.0a) and one experiment with wave-state-dependent flux (COR3.0a-WAV) were conducted. We modify COARE3.0a to include surface roughness calculated within WW3 and also account for the buffering effect of waves on the relative difference between air-side and ocean-side momentum flux. Differences in surface fluxes, primarily caused by discrepancies in drag coefficients, result in significant differences in MJO’s properties. While COARE3.0a has better convection–circulation coupling than NCAR, it exhibits anomalous MJO convection east of the date line. The wave-state-dependent flux (COR3.0-WAV) improves the MJO representation over the default COARE3.0 algorithm. Strong easterlies over the Pacific Ocean in COARE3.0a enhance the latent heat flux (LHFLX). This is responsible for the anomalous MJO propagation after the date line. In COR3.0a-WAV, waves reduce the anomalous easterlies, leading to a decrease in LHFLX and MJO dissipation after the date line. These findings highlight the role of surface fluxes in MJO simulation fidelity. Most importantly, we show that the proper treatment of wave-induced effects in bulk flux parameterization improves the simulation of coupled climate variability.

54 ENVIRONMENTAL SCIENCES↗

Extrapolar Cloud Feedbacks as a Driver of Arctic Amplification

The role of cloud feedbacks in Arctic amplification (AA) of anthropogenic warming remains unclear. Traditional feedback analysis diagnoses the net cloud feedback as strongly positive in the tropics but either weak or negative in the Arctic, suggesting that AA would be amplified if cloud feedbacks were suppressed. However, in cloud-locking experiments using the slab ocean version of the Energy Exascale Earth System Model (E3SM), we find that suppressing cloud feedbacks results in a substantial decrease in AA under greenhouse gas forcing. We show that the increase in AA from cloud feedbacks arises from two main mechanisms: 1) the additional energy contributed by positive cloud feedbacks in the tropics leads to increased poleward moist atmospheric heat transport (AHT) which then amplifies Arctic warming; and 2) the additional Arctic warming is amplified by positive noncloud feedbacks in the region, together making extrapolar cloud feedbacks amplify AA. We also find that cloud changes can modify the strength of noncloud feedback, but that modification has a small effect on Arctic warming. We further examine the role of cloud feedbacks in AA using a moist energy balance model, which demonstrates that interactions of cloud feedbacks with moist AHT and other positive feedbacks dominate the influence of clouds on the pattern of surface warming. However, the contribution of cloud-induced changes in noncloud feedbacks on AA is relatively minor. Furthermore, these results demonstrate that traditional attributions of AA, that are based on local feedback analysis, overlook key interactions between extrapolar cloud changes, poleward AHT, and noncloud feedbacks in the Arctic.

58 GEOSCIENCES↗

Examining Cloud Feedback Components in the Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM)

Cloud feedback remains the main source of uncertainty in climate sensitivity estimated by global climate models (GCMs), largely because subgrid cloud responses are parameterized in GCMs due to their coarse resolution. Here, this study examines cloud feedback in the global 3.25-km Simple Cloud-Resolving Energy Exascale Earth System Model (E3SM) Atmosphere Model (SCREAM 3 km) through a pair of 1-yr atmosphere-only simulations with control and +4-K sea surface temperature perturbations. SCREAM 3 km produces a positive cloud feedback that falls within but at the upper end of the range of Coupled Model Intercomparison Project phase 5 (CMIP5) and CMIP phase 6 (CMIP6) models and expert judgment. The positive cloud feedback arises from positive contributions from both high- and low-level clouds, with increases in high-cloud altitude and decreases in low-cloud amount and optical depth playing key roles. The stronger-than-CMIP-average feedback is mainly attributable to the high-cloud altitude feedback, owing to cloud tops rising nearly isothermally in SCREAM 3 km. The positive low-cloud amount feedback is weaker in SCREAM than in GCMs because estimated inversion strength (EIS) increases more dramatically with warming. A coarser 12-km resolution version of SCREAM exhibits a weaker positive cloud feedback than SCREAM 3 km, mainly because its low-cloud-radiative flux is more sensitive to EIS, leading to a stronger negative low-cloud amount feedback. With this process-level assessment of cloud feedback, this study reveals where SCREAM aligns with and diverges from conventional GCMs and expert assessment, providing insights to inform further model improvement and future expert assessment.

Cloud radiative effects↗

Land–Atmosphere Coupling at the U.S. Southern Great Plains: A Comparison on Local Convective Regimes between ARM Observations, Reanalysis, and Climate Model Simulations

Using the 9-yr warm-season observations at the Atmospheric Radiation Measurement Southern Great Plains site, in this work we assess the land–atmosphere (LA) coupling in the North American Regional Reanalysis (NARR) and two climate models: hindcasts with the Community Atmosphere Model version 5.1 by Cloud-Associated Parameterizations Testbed (CAM5-CAPT) and nudged runs with the Energy Exascale Earth System Model Atmosphere Model version 1 Regionally Refined Model (EAMv1-RRM). We focus on three local convective regimes and diagnose model behaviors using the local coupling metrics. NARR agrees well with observations except a slightly warmer and drier surface with higher downwelling shortwave radiation and lower evaporative fraction. On clear-sky days, it shows warmer and drier early-morning conditions in both models with significant underestimates in surface evaporation by EAMv1-RRM. On the majority of the ARM-observed shallow cumulus days, there is no or little low-level clouds in either model. When captured in models, the simulated shallow cumulus shows much less cloud fraction and lower cloud bases than observed. On the days with late-afternoon deep convection, models tend to present a stable early-morning lower atmosphere more frequently than the observations, suggesting that the deep convection is triggered more often by elevated instabilities. Generally, CAM5-CAPT can reproduce the local LA coupling processes to some extent due to the constrained early-morning conditions and large-scale winds. EAMv1-RRM exhibits large precipitation deficits and warm and dry biases toward mid-to-late summers, which may be an amplification through a positive LA feedback among initial atmosphere and land states, convection triggering and large-scale circulations.

58 GEOSCIENCES↗

Quantifying the Impacts of Land-Cover Change on the Hydrologic Response to Hurricane Ida in the Lower Mississippi River Basin

Abstract The Lower Mississippi River basin (LMRB) has experienced significant changes in land cover and is one of the most vulnerable regions to hurricanes in the United States. Here, we study the impacts of land-cover change on the hydrologic response to Hurricane Ida in LMRB. By using an integrated surface–subsurface hydrologic model, Energy Exascale Earth System Model (E3SM) Land Model coupled with the three-dimensional ParFlow subsurface flow model (ELM-ParFlow), we simulate the effects of land-cover change on the flood volume and peak timing induced by rainfall from Hurricane Ida. The results show that land-cover changes from 1850 to 2015, which resulted in a smoother surface and less vegetation, exacerbated both flood peak time and volume induced by Hurricane Ida. The effects of land-cover changes can be decomposed into two mechanisms: a smoother surface routes more water faster to a watershed outlet and less vegetation allows more water to contribute to surface runoff. By comparing scenarios in which the two mechanisms were isolated, we found that changes in soil moisture due to vegetation cover change have more dominant effects on floods in the southern part and changes in Manning’s coefficient have the largest effect on floods in the northern part of the LMRB. The study provides important insights into the complex relationship between land-use, land-cover, and hydrologic processes in coastal regions.

54 ENVIRONMENTAL SCIENCES↗

Physics-Dynamics Coupling in Weather, Climate, and Earth System Models: Challenges and Recent Progress

Geophysical models of the atmosphere and ocean invariably involve parameterizations. These represent two distinct areas: a) Subgrid processes which the model cannot (yet) resolve, due to its discrete resolution, and b) sources in the equation, due to radiation for example. Hence coupling between these physics parameterizations and the resolved fluid dynamics and also between the dynamics of the different fluids in the system (air and water) is necessary. This coupling is an important aspect of geophysical models. However, often model development is strictly segregated into either physics or dynamics. Hence, this area has many more unanswered questions than in-depth understanding. Furthermore, recent developments in the design of dynamical cores (significant increase of resolution, move to non-hydrostatic equation sets etc), extended process physics (prognostic micro physics, 3D turbulence, non-vertical radiation etc) and predicted future changes of the computational infrastructure (Exascale with its need for task parallelism, data locality and asynchronous time stepping for example) is adding even more complexity and new questions.

Gross, Markus↗

Assessment of Precipitating Marine Stratocumulus Clouds in the E3SMv1 Atmosphere Model: A Case Study from the ARM MAGIC Field Campaign

This paper presents a process-oriented evaluation of precipitating stratocumulus and its transition to cumulus in version 1 of the Energy Exascale Earth System Model (E3SMv1) using comprehensive case-study observations from a field campaign of the Atmospheric Radiation Measurement program (ARM). The E3SMv1 single-column model (SCM) of the marine boundary layer and its low clouds and precipitation are compared to observations including subcloud drizzle retrievals from a combination of Doppler radar and lidar backscatter measurements. Additionally, the SCM is also compared to a large-eddy simulation (LES) of the same case. The combination of advanced remote sensing observations and LES is a powerful framework to evaluate the physical parameterizations of large-scale models. Given the observed large-scale environment, the E3SMv1 SCM realistically represents the evolution of clouds and boundary layer structure during the stratocumulus-to-cumulus transition. The model well simulates the liquid water path and its diurnal cycle in the stratocumulus period as well as the two-layer vertical thermodynamic structure and lower cloud fraction in the transition period. E3SMv1’s success in simulating the cloud in the stratocumulus period permitted examination of its precipitation processes. Here problems were identified with E3SMv1 producing an unrealistically small subcloud precipitation fraction, an unrealistic double peak in the vertical profiles of precipitation mass, and drizzle that evaporates too close to the surface. Further model diagnostics determined that these unrealistic characteristics resulted from an overly long microphysics time step and an unrealistic parameterization of the precipitation fraction. These results imply that careful consideration of these issues is needed in order to better simulate precipitation processes in marine stratocumulus.

54 ENVIRONMENTAL SCIENCES↗

PANORAMA: An approach to performance modeling and diagnosis of extreme-scale workflows

Here we report that computational science is well established as the third pillar of scientific discovery and is on par with experimentation and theory. However, as we move closer toward the ability to execute exascale calculations and process the ensuing extreme-scale amounts of data produced by both experiments and computations alike, the complexity of managing the compute and data analysis tasks has grown beyond the capabilities of domain scientists. Therefore, workflow management systems are absolutely necessary to ensure current and future scientific discoveries. A key research question for these workflow management systems concerns the performance optimization of complex calculation and data analysis tasks. The central contribution of this article is a description of the PANORAMA approach for modeling and diagnosing the run-time performance of complex scientific workflows. This approach integrates extreme-scale systems testbed experimentation, structured analytical modeling, and parallel systems simulation into a comprehensive workflow framework called Pegasus for understanding and improving the overall performance of complex scientific workflows.

97 MATHEMATICS AND COMPUTING↗

A massively parallel and memory-efficient FEM toolbox with a hybrid total FETI solver with accelerator support

In this article, we present the ExaScale PaRallel finite element tearing and interconnecting SOlver (ESPRESO) finite element method (FEM) library, which includes an FEM toolbox with interfaces to professional and open-source simulation tools, and a massively parallel hybrid total finite element tearing and interconnecting (HTFETI) solver which can fully utilize the Oak Ridge Leadership Computing Facility Titan supercomputer and achieve superlinear scaling. This article presents several new techniques for finite element tearing and interconnecting (FETI) solvers designed for efficient utilization of supercomputers with a focus on (i) performance—we present a fivefold reduction of solver runtime for the Laplace equation by redesigning the FETI solver and offloading the key workload to the accelerator. We compare Intel Xeon Phi 7120p and Tesla K80 and P100 accelerators to Intel Xeon E5-2680v3 and Xeon Phi 7210 central processing units; and (ii) memory efficiency—we present two techniques which increase the efficiency of the HTFETI solver 1.8 times and push the limits of the largest possible problem ESPRESO that can solve from 124 to 223 billion unknowns for problems with unstructured meshes. Finally, we show that by dynamically tuning hardware parameters, we can reduce energy consumption by up to 33%.

Computer Science↗

Harnessing billions of tasks for a scalable portable hydrodynamic simulation of the merger of two stars

We propose a highly scalable demonstration of a portable asynchronous many-task programming model and runtime system applied to a grid-based adaptive mesh refinement hydrodynamic simulation of a double white dwarf merger with 14 levels of refinement that spans 17 orders of magnitude in astrophysical densities. The code uses the portable C++ parallel programming model that is embodied in the HPX library and being incorporated into the ISO C++ standard. The model reflects a significant shift from existing bulk synchronous parallel programming models under consideration for exascale systems. Through the use of the Futurization technique, seemingly sequential code is transformed into wait-free asynchronous tasks. We demonstrate the potential of our model by showing results from strong scaling runs on National Energy Research Scientific Computing Center’s Cori system (658,784 Intel Knight’s Landing cores) that achieve a parallel efficiency of 96.8% using billions of asynchronous tasks.

79 ASTRONOMY AND ASTROPHYSICS↗

Computational Reproducibility of Scientific Workflows at Extreme Scales

We propose an approach for improved reproducibility that includes capturing and relating provenance characteristics and performance metrics, in a hybrid queriable system, the ProvEn server. The system capabilities are illustrated on two use cases: scientific reproducibility of results in the ACME climate simulations and performance reproducibility in molecular dynamics workflows on HPC computing platforms.

provenance, Reproducibility, Climate & Earth Syste↗

Evaluating asynchronous Schwarz solvers on GPUs

With the commencement of the exascale computing era, we realize that the majority of the leadership supercomputers are heterogeneous and massively parallel. Even a single node can contain multiple co-processors such as GPUs and multiple CPU cores. For example, ORNL’s Summit accumulates six NVIDIA Tesla V100 GPUs and 42 IBM Power9 cores on each node. Synchronizing across compute resources of multiple nodes can be prohibitively expensive. Hence, it is necessary to develop and study asynchronous algorithms that circumvent this issue of bulk-synchronous computing. In this study, we examine the asynchronous version of the abstract Restricted Additive Schwarz method as a solver. We do not explicitly synchronize, but allow the communication between the sub-domains to be completely asynchronous, thereby removing the bulk synchronous nature of the algorithm. We accomplish this by using the one-sided Remote Memory Access (RMA) functions of the MPI standard. We study the benefits of using such an asynchronous solver over its synchronous counterpart. We also study the communication patterns governed by the partitioning and the overlap between the sub-domains on the global solver. Finally, we show that this concept can render attractive performance benefits over the synchronous counterparts even for a well-balanced problem.

Nayak, Pratik↗

Unprecedented cloud resolution in a GPU-enabled full-physics atmospheric climate simulation on OLCF’s summit supercomputer

Clouds represent a key uncertainty in future climate projection. While explicit cloud resolution remains beyond our computational grasp for global climate, we can incorporate important cloud effects through a computational middle ground called the Multi-scale Modeling Framework (MMF), also known as Super Parameterization. This algorithmic approach embeds high-resolution Cloud Resolving Models (CRMs) to represent moist convective processes within each grid column in a Global Climate Model (GCM). The MMF code requires no parallel data transfers and provides a self-contained target for acceleration. This study investigates the performance of the Energy Exascale Earth System Model-MMF (E3SM-MMF) code on the OLCF Summit supercomputer at an unprecedented scale of simulation. Hundreds of kernels in the roughly 10K lines of code in the E3SM-MMF CRM were ported to GPUs with OpenACC directives. A high-resolution benchmark using 4600 nodes on Summit demonstrates the computational capability of the GPU-enabled E3SM-MMF code in a full physics climate simulation.

58 GEOSCIENCES↗

Resilience and fault tolerance in high-performance computing for numerical weather and climate prediction

Progress in numerical weather and climate prediction accuracy greatly depends on the growth of the available computing power. As the number of cores in top computing facilities pushes into the millions, increased average frequency of hardware and software failures forces users to review their algorithms and systems in order to protect simulations from breakdown. This report surveys hardware, application-level and algorithm-level resilience approaches of particular relevance to time-critical numerical weather and climate prediction systems. A selection of applicable existing strategies is analysed, featuring interpolation-restart and compressed checkpointing for the numerical schemes, in-memory checkpointing, user-level failure mitigation and backup-based methods for the systems. Numerical examples showcase the performance of the techniques in addressing faults, with particular emphasis on iterative solvers for linear systems, a staple of atmospheric fluid flow solvers. The potential impact of these strategies is discussed in relation to current development of numerical weather prediction algorithms and systems towards the exascale. Trade-offs between performance, efficiency and effectiveness of resiliency strategies are analysed and some recommendations outlined for future developments.

54 ENVIRONMENTAL SCIENCES↗

Performance of explicit and IMEX MRI multirate methods on complex reactive flow problems within modern parallel adaptive structured grid frameworks

Large-scale multiphysics simulations are computationally challenging due to the coupling of multiple processes with widely disparate time scales. The advent of exascale computing systems exacerbates these challenges since these systems enable ever-increasing size and complexity. In recent years, there has been renewed interest in developing multirate methods as a means to handle the large range of time scales, as these methods may afford greater accuracy and efficiency than more traditional approaches of using implicit-explicit (IMEX) and low-order operator splitting schemes. However, to date there have been few performance studies that compare different classes of multirate integrators on complex application problems. In this work, we study the performance of several newly developed multirate infinitesimal (MRI) methods, implemented in the SUNDIALS solver package, on two reacting flow model problems built on structured mesh frameworks. The first model revisits prior work on a compressible reacting flow problem with complex chemistry that is implemented using BoxLib but where we now include comparisons between a new explicit MRI scheme with the multirate spectral deferred correction (SDC) methods in the original paper. The second problem uses the same complex chemistry as the first problem, combined with a simplified flow model, but runs at a large spatial scale where explicit methods become infeasible due to stability constraints. Two recently developed IMEX MRI multirate methods are tested. These methods rely on advanced features of the AMReX framework on which the model is built, such as multilevel grids and multilevel preconditioners. The results from these two problems show that MRI multirate methods can offer significant performance benefits on complex multiphysics application problems and that these methods may be combined with advanced spatial discretization to compound the advantages of both.

97 MATHEMATICS AND COMPUTING↗

A graphics processing unit accelerated sparse direct solver and preconditioner with block low rank compression

We present the GPU implementation efforts and challenges of the sparse solver package STRUMPACK. The code is made publicly available on github with a permissive BSD license. STRUMPACK implements an approximate multifrontal solver, a sparse LU factorization which makes use of compression methods to accelerate time to solution and reduce memory usage. Multiple compression schemes based on rank-structured and hierarchical matrix approximations are supported, including hierarchically semi-separable, hierarchically off-diagonal butterfly, and block low rank. Here, in this paper, we present the GPU implementation of the block low rank (BLR) compression method within a multifrontal solver. Our GPU implementation relies on highly optimized vendor libraries such as cuBLAS and cuSOLVER for NVIDIA GPUs, rocBLAS and rocSOLVER for AMD GPUs and the Intel oneAPI Math Kernel Library (oneMKL) for Intel GPUs. Additionally, we rely on external open source libraries such as SLATE (Software for Linear Algebra Targeting Exascale), MAGMA (Matrix Algebra on GPU and Multi-core Architectures), and KBLAS (KAUST BLAS). SLATE is used as a GPU-capable ScaLAPACK replacement. From MAGMA we use variable sized batched dense linear algebra operations such as GEMM, TRSM and LU with partial pivoting. KBLAS provides efficient (batched) low rank matrix compression for NVIDIA GPUs using an adaptive randomized sampling scheme. The resulting sparse solver and preconditioner runs on NVIDIA, AMD and Intel GPUs. Interfaces are available from PETSc, Trilinos and MFEM, or the solver can be used directly in user code. We report results for a range of benchmark applications, using the Perlmutter system from NERSC, Frontier from ORNL, and Aurora from ALCF. For a high frequency wave equation on a regular mesh, using 32 Perlmutter compute nodes, the factorization phase of the exact GPU solver is about 6.5× faster compared to the CPU-only solver. The BLR-enabled GPU solver is about 13.8× faster than the CPU exact solver. For a collection of SuiteSparse matrices, the STRUMPACK exact factorization on a single GPU is on average 1.9× faster than NVIDIA’s cuDSS solver.

97 MATHEMATICS AND COMPUTING↗

PaRSEC: Scalability, flexibility, and hybrid architecture support for task-based applications in ECP

This paper highlights the most significant enhancements made to PaRSEC, a scalable task-based runtime system designed for hybrid machines, during the Exascale Computing Project (ECP). The enhancements focus on expanding the capabilities of PaRSEC to address the evolving landscape of parallel computing. Notable achievements include the integration of support for three major types of accelerators (NVIDIA, AMD, and Intel GPUs), the refinement and increased flexibility of the communication subsystem, and the introduction of new programming interfaces tailored for irregular applications. Additionally, the project resulted in the development of powerful debugging and performance analysis tools aimed at assisting users in understanding and optimizing their applications. We present a comprehensive demonstration of these advancements through a series of benchmarks and applications within ECP and beyond, thereby showcasing the enhanced capabilities of PaRSEC across the diverse architectures within the ECP, providing valuable insights into the runtime system’s adaptability and performance across varied computing environments.

Bouteiller, Aurelien↗