Exascale Computing Project Post-Mortem
Content is mostly from an external Exascale Project worked on prior to joining Sandia.
SEARCH · Engineering Papers
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Content is mostly from an external Exascale Project worked on prior to joining Sandia.
Partitioned Global Address Space (PGAS) models, typified by such languages as Unified Parallel C (UPC) and Co-Array Fortran, expose one-sided communication as a key building block for High Performance Computing (HPC) applications. Architectural trends in supercomputing make such programming models increasingly attractive, and newer, more sophisticated models such as UPC++, Legion and Chapel that rely upon similar communication paradigms are gaining popularity. GASNet-EX is a portable, open-source, high-performance communication library designed to efficiently support the networking requirements of PGAS runtime systems and other alternative models in future exascale machines. The library is an evolution of the popular GASNet communication system, building upon over 15 years of lessons learned. We describe and evaluate several features and enhancements that have been introduced to address the needs of modern client systems. Microbenchmark results demonstrate the RMA performance of GASNet-EX is competitive with several MPI-3 implementations on current HPC systems.
We present ExaTN (Exascale Tensor Networks), a scalable GPU-accelerated C++ library which can express and process tensor networks on shared- as well as distributed-memory high-performance computing platforms, including those equipped with GPU accelerators. Specifically, ExaTN provides the ability to build, transform, and numerically evaluate tensor networks with arbitrary graph structures and complexity. It also provides algorithmic primitives for the optimization of tensor factors inside a given tensor network in order to find an extremum of a chosen tensor network functional, which is one of the key numerical procedures in quantum many-body theory and quantum-inspired machine learning. Numerical primitives exposed by ExaTN provide the foundation for composing rather complex tensor network algorithms. We enumerate multiple application domains which can benefit from the capabilities of our library, including condensed matter physics, quantum chemistry, quantum circuit simulations, as well as quantum and classical machine learning, for some of which we provide preliminary demonstrations and performance benchmarks just to emphasize a broad utility of our library.
The application of data science in cancer research has been boosted by major advances in three primary areas: (1) Data: diversity, amount, and availability of biomedical data; (2) Advances in Artificial Intelligence (AI) and Machine Learning (ML) algorithms that enable learning from complex, large-scale data; and (3) Advances in computer architectures allowing unprecedented acceleration of simulation and machine learning algorithms. These advances help build in silico ML models that can provide transformative insights from data including: molecular dynamics simulations, next-generation sequencing, omics, imaging, and unstructured clinical text documents. Unique challenges persist, however, in building ML models related to cancer, including: (1) access, sharing, labeling, and integration of multimodal and multi-institutional data across different cancer types; (2) developing AI models for cancer research capable of scaling on next generation high performance computers; and (3) assessing robustness and reliability in the AI models. In this paper, we review the National Cancer Institute (NCI) -Department of Energy (DOE) collaboration, Joint Design of Advanced Computing Solutions for Cancer (JDACS4C), a multi-institution collaborative effort focused on advancing computing and data technologies to accelerate cancer research on three levels: molecular, cellular, and population. This collaboration integrates various types of generated data, pre-exascale compute resources, and advances in ML models to increase understanding of basic cancer biology, identify promising new treatment options, predict outcomes, and eventually prescribe specialized treatments for patients with cancer.
Abstract We present an extension of the massively parallel, GPU native, astrophysical hydrodynamics code Cholla to magnetohydrodynamics (MHD). Cholla solves the ideal MHD equations in their Eulerian form on a static Cartesian mesh utilizing the Van Leer + constrained transport integrator, the HLLD Riemann solver, and reconstruction methods at second and third order. Cholla’s MHD module can perform ≈260 million cell updates per GPU-second on an NVIDIA A100 while using the HLLD Riemann solver and second order reconstruction. The inherently parallel nature of GPUs combined with increased memory in new hardware allows Cholla’s MHD module to perform simulations with resolutions ∼500 3 cells on a single high-end GPU (e.g., an NVIDIA A100 with 80 GB of memory). We employ GPU direct Message Passing Interface to attain excellent weak scaling on the exascale supercomputer Frontier, while using 74,088 GPUs and simulating a total grid size of over 7.2 trillion cells. A suite of test problems highlights the accuracy of Cholla’s MHD module and demonstrates that zero magnetic divergence in solutions is maintained to round off error. We also present new testing and CI tools using GoogleTest, GitHub Actions, and Jenkins that have made development more robust and accurate and ensure reliability in the future.
This study investigates aerosol-cloud interactions in marine boundary layer (MBL) clouds using an advanced deep-learning-driven synoptic-regime-based framework, combining satellite data (CALIPSO vertically resolved aerosol extinction and MODIS cloud properties) with 1° nudged Energy Exascale Earth System Model version 2 (E3SMv2) simulation over the Eastern North Atlantic (ENA; ∼10°×10°, 2006–2014). The E3SMv2 captures observed seasonal variations in cloud droplet number concentrations (N d ) and liquid water path (LWP), though it systematically underestimates N d . We then partition ENA meteorology into four synoptic regimes (Pre-Trough, Post-Trough, Ridge, Trough) via a deep-learning clustering of ERA5 reanalysis fields, enabling regime-dependent aerosol-cloud interactions analyses. Both satellite and E3SMv2 exhibit an inverted-V LWP-N d relationship. In Post-Trough and Ridge regimes, the satellite shows stronger negative LWP-N d sensitivities than in Pre-Trough regime. The Trough regime displays a muted satellite LWP response. In comparison, the model predicts more exaggerated LWP responses across regimes, with LWP increasing too quickly at low N d and decreasing more sharply at high N d , especially in Pre-Trough and Trough regimes. These exaggerated model LWP sensitivities may stem from uncertainties in representing drizzle processes, entrainment, and turbulent mixing. As for N d susceptibility to aerosols, N d increases with MBL aerosol extinction in both datasets, but the simulated aerosol-cloud interactions appear oversensitive to meteorological conditions. Overall, E3SMv2 better captures aerosol effects under regimes that favor stratiform clouds (Post-Trough, Ridge), but performance deteriorates for regimes with deeper, dynamically complex clouds (Trough), highlighting the need for improved representations of those cloud processes in climate models.
Abstract. The importance of carbon (C)–nutrient interactions to the prediction of future C uptake has long been recognized. The Energy Exascale Earth System Model (E3SM) land model (ELM) version 1 is one of the few land surface models that include both N and P cycling and limitation (ELMv1-CNP). Here we provide a global-scale evaluation of ELMv1-CNP using the International Land Model Benchmarking (ILAMB) system. We show that ELMv1-CNP produces realistic estimates of present-day carbon pools and fluxes. Compared to simulations with optimal P availability, simulations with ELMv1-CNP produce better performance, particularly for simulated biomass, leaf area index (LAI), and global net C balance. We also show ELMv1-CNP-simulated N and P cycling is in good agreement with data-driven estimates. We compared the ELMv1-CNP-simulated response to CO2 enrichment with meta-analysis of observations from similar manipulation experiments. We show that ELMv1-CNP is able to capture the field-observed responses for photosynthesis, growth, and LAI. We investigated the role of P limitation in the historical balance and show that global C sources and sinks are significantly affected by P limitation, as the historical CO2 fertilization effect was reduced by 20 % and C emission due to land use and land cover change was 11 % lower when P limitation was considered. Our simulations suggest that the introduction of P cycle dynamics and C–N–P coupling will likely have substantial consequences for projections of future C uptake.
Abstract. We present an architecture-portable and performant implementation of the atmospheric dynamical core (High-Order Methods Modeling Environment, HOMME) of the Energy Exascale Earth System Model (E3SM). The original Fortran implementation is highly performant and scalable on conventional architectures using the Message Passing Interface(MPI) and Open MultiProcessor (OpenMP) programming models.We rewrite the model in C++ and use the Kokkos library to express on-node parallelism in a largely architecture-independent implementation. Kokkos provides an abstraction of a compute node or device,layout-polymorphic multidimensional arrays, and parallel execution constructs. The new implementation achieves the same or better performance on conventional multicore computers and is portable to GPUs. We present performance data for the original and new implementations on multiple platforms, on up to 5400 compute nodes, and study several aspects of the single- and multi-node performance characteristics of the new implementation on conventional CPU (e.g., Intel Xeon), many core CPU (e.g., Intel Xeon Phi Knights Landing),and Nvidia V100 GPU.
Wind-wave processes have generally been excluded from coupled Earth system models due to the high computational expense of spectral wave models, which resolve a frequency and direction spectrum of waves across space and time. Existing uniform-resolution wave modeling approaches used in Earth system models cannot appropriately represent wave climates from global to coastal ocean scales, largely because of tradeoffs between coastal resolution and computational costs. To resolve this challenge, we introduce a global unstructured mesh capability for the WAVEWATCH III (WW3) model that is suitable for coupling within the US Department of Energy's Energy Exascale Earth System Model (E3SM). The new unstructured WW3 global wave modeling approach can provide the accuracy of higher global resolutions in coastal areas at the relative cost of lower uniform global resolutions. This new capability enables simulation of waves at physically relevant scales as needed for coastal applications.
Abstract. The Energy Exascale Earth System Model (E3SM) developed by the Department of Energy has a goal of addressing challenges in understanding the global water cycle. Success depends on correct simulation of cloud and precipitation elements. However, lack of appropriate evaluation metrics has hindered the accurate representation of these elements in general circulation models. We derive metrics from the three-dimensional data of the ground-based Next-Generation Radar (NEXRAD) network over the US to evaluate both horizontal and vertical structures of precipitation elements. We coarsened the resolution of the radar observations to be consistent with the model resolution and improved the coupling of the Cloud Feedback Model Intercomparison Project Observation Simulator Package (COSP) and E3SM Atmospheric Model Version 1 (EAMv1) to obtain the best possible model output for comparison with the observations. Three warm seasons (2014–2016) of EAMv1 simulations of 3-D radar reflectivity features at an hourly scale are evaluated. A general agreement in domain-mean radar reflectivity intensity is found between EAMv1 and NEXRAD below 4 km altitude; however, the model underestimates reflectivity over the central US, which suggests that the model does not capture the mesoscale convective systems that produce much of the precipitation in that region. The shape of the model-estimated histogram of subgrid-scale reflectivity is improved by correcting the microphysical assumptions in COSP. Different from previous studies that evaluated modeled cloud top height, we find the model severely underestimates radar reflectivity at upper levels – the simulated echo top height is about 5 km lower than in observations – and this result is not changed by tuning any single physics parameter. For more accurate model evaluation, a higher-order consistency between the COSP and the host model is warranted in future studies.
Abstract. Runoff is a critical component of the terrestrial water cycle, and Earth system models (ESMs) are essential tools to study its spatiotemporal variability. Runoff schemes in ESMs typically include many parameters so that model calibration is necessary to improve the accuracy of simulated runoff. However, runoff calibration at a global scale is challenging because of the high computational cost and the lack of reliable observational datasets. In this study, we calibrated 11 runoff relevant parameters in the Energy Exascale Earth System Model (E3SM) Land Model (ELM) using a surrogate-assisted Bayesian framework. First, the polynomial chaos expansion machinery with Bayesian compressed sensing is used to construct computationally inexpensive surrogate models for ELM-simulated runoff at 0.5∘ × 0.5∘ for 1991–2010. The error metric between the ELM simulations and the benchmark data is selected to construct the surrogates, which facilitates efficient calibration and avoids the more conventional, but challenging, construction of high-dimensional surrogates for the ELM simulated runoff. Second, the Sobol' index sensitivity analysis is performed using the surrogate models to identify the most sensitive parameters, and our results show that, in most regions, ELM-simulated runoff is strongly sensitive to 3 of the 11 uncertain parameters. Third, a Bayesian method is used to infer the optimal values of the most sensitive parameters using an observation-based global runoff dataset as the benchmark. Our results show that model performance is significantly improved with the inferred parameter values. Although the parametric uncertainty of simulated runoff is reduced after the parameter inference, it remains comparable to the multimodel ensemble uncertainty represented by the global hydrological models in ISMIP2a. Additionally, the annual global runoff trend during the simulation period is not well constrained by the inferred parameter values, suggesting the importance of including parametric uncertainty in future runoff projections.
Abstract. The Energy Exascale Earth System Model (E3SM) project is an ongoing, state-of-the-science Earth system modeling, simulation, and prediction project developed by the US Department of Energy (DOE). With an emphasis on supporting the DOE's energy mission, understanding and quantifying how well the model simulates water cycle processes is of particular importance. Here, we evaluate E3SM version 1.0 (v1.0) for its ability to represent atmospheric rivers (ARs), which play significant roles in water vapor transport and precipitation. The characteristics and precipitation associated with global ARs in E3SM at standard resolution (1∘ × 1∘) are compared to the Modern-Era Retrospective analysis for Research and Applications, version 2 (MERRA2). Global patterns of AR frequencies in E3SM show high degrees of correlation (≥0.97) with MERRA2 and low mean absolute errors (MAEs; <1 %) annually, seasonally, and across different ensemble members. However, some large-scale condition biases exist, leading to AR biases – most significant of which are the double intertropical convergence zone (ITCZ), a stronger and/or equatorward-shifted subtropical jet during boreal and austral winters, and enhanced Northern Hemisphere westerlies during summer. By comparing atmosphere-only and fully coupled simulations, we attribute the sources of the biases to the atmospheric component or to a coupling response. Using relationships revealed in Dong et al. (2021), we provide evidence showing the stronger North Pacific jet in winter and the enhanced Northern Hemisphere westerlies during summer, associated with E3SM's double ITCZ and related weaker Atlantic meridional overturning circulation (AMOC), respectively, which are significant sources of the AR biases found in the coupled simulations.
Abstract. Streamflow variability plays a crucial role in shaping the dynamics and sustainability of Earth's ecosystems, which can be simulated and projected by a river routing model coupled with a land surface model. However, the simulation of streamflow at large scales is subject to considerable uncertainties, primarily arising from two related processes: runoff generation (hydrological process) and river routing (hydraulic process). While both processes have impacts on streamflow variability, previous studies only calibrated one of the two processes to reduce biases in the simulated streamflow. Calibration focusing only on one process can result in unrealistic parameter values to compensate for the bias resulting from the other process; thus other water-related variables remain poorly simulated. In this study, we performed several experiments with the land and river components of the Energy Exascale Earth System Model (E3SM) over the Pantanal region to disentangle the hydrological and hydraulic controls on streamflow variability in coupled land–river simulations. Our results show that the generation of subsurface runoff is the most important factor for streamflow variability contributed by the runoff generation process, while floodplain storage effect and main-channel roughness have significant impacts on streamflow variability through the river routing process. We further propose a two-step procedure to robustly calibrate the two processes together. The impacts of runoff generation and river routing on streamflow are appropriately addressed with the two-step calibration, which may be adopted by developers of land surface and earth system models to improve the modeling of streamflow.
Abstract. The interaction between clouds and radiation is a key process within the climate system, and assessing the impacts of that interaction provides valuable insights into both the present-day climate and future projections. Many modeling experiments have been designed over the years to probe the impact of the cloud radiative effect (CRE) on the climate, including those that seek to disrupt the mean CRE effect and those that only disrupt the covariance of the CRE with the circulation. Seven such experimental designs have been added to the Energy Exascale Earth System Model version 1 (E3SMv1) of the US Department of Energy. These experiments include both the first and second iterations of the Clouds On/Off Klimate Intercomparison Experiment (COOKIE) experimental design, as well as the cloud-locking method. This paper documents the code changes necessary to implement such experiments and also provides detailed instructions for how to run them. Analyses across experiment types provide valuable insights and confirm the findings of prior studies, including the role of cloud radiative heating toward intensifying the monsoon, intensifying rain rates, and poleward expansion of the general circulation owing to cloud feedbacks.
When the nutrient level in the soil surpasses vegetation demand, nutrient losses due to surface runoff and subsurface leaching are the major reasons for the deterioration of water quality. The lower Mississippi River basin (LMRB) is one of the sub-basins that deliver the highest nitrogen loads to the Gulf of Mexico. Potential changes in episodic events induced by hurricanes may exacerbate water quality issue in the future. However, uncertainties in modeling the hydrologic response to hurricanes may limit the modeling of nutrient losses during such events. Using a machine learning approach, we calibrated the land component of the Energy Exascale Earth System Model (E3SM), or ELM, version 2.1, based on the water table depth (WTD) of a calibrated 3D subsurface hydrology model. While the overall performance of the calibrated ELM is satisfactory, some discrepancies in WTD remain in slope areas with low precipitation due to the missing lateral flow process in ELM. Simulations including biogeochemistry performed using ELM with and without model calibration showed important influences of soil hydrology, precipitation intensity, and runoff parameterization on the magnitude of nitrogen runoff loss and the leaching pathway. Despite such sensitivities, both ELM simulations produced reduced WTD and increased runoff and accelerated nitrate–nitrogen runoff loading during Hurricane Ida in August 2021, consistent with the observations. With observations suggesting more pronounced effects of Hurricane Ida on nitrogen runoff than the simulations, we identified factors for model improvement to provide a useful tool for studying hurricane-induced nutrient losses in the LMRB region.
Abstract. We present an architecture-portable and performant implementation of the atmospheric dynamical core (HOMME) of the Energy Exascale Earth System Model (E3SM). The original Fortran implementation is highly performant and scalable on conventional architectures using MPI and OpenMP. We rewrite the model in C++ and use the Kokkos library to express on-node parallelism in a largely architecture-independent implementation. Kokkos provides an abstraction of a compute node or device, layout-polymorphic multidimensional arrays, and parallel execution constructs. Here, the new implementation achieves the same or better performance on conventional multicore computers and is portable to GPUs. We present performance data for the original and new implementations on multiple platforms, on up to 5400 compute nodes, and study several aspects of the single- and multi-node performance characteristics of the new implementation on conventional CPU, Intel Xeon Phi Knights Landing, and Nvidia V100 GPU.
Summary of technical results of Blackcomb Memory Devices We explored various different memory technologies (STTRAM, PCRAM, FeRAM, and ReRAM). The progress can be classified into three categories, below. Modeling and Tool Releases Various modeling tools have been developed over the last decade to help in the design of SRAM or DRAM-based memory hierarchies. To explore new design opportunities that NVM technologies can bring to the designers, we have developed similar high-level models for NVM, including PCRAMsim [Dong 2009], NVSim [Dong 2012], and NVMain [Poremba 2012]. NVSim is a circuit-level model for NVM performance, energy, and area estimation, which supports various NVM technologies, including STT-RAM, PCRAM, ReRAM, and legacy NAND Flash. NVSim is successfully validated against industrial NVM prototypes, and it is expected to help boost architecture-level NVM-related studies. On the other side, NVMain is a cycle accurate main memory simulator designed to simulate emerging nonvolatile memories at the architectural level. We have released these models as open source tools and provided contiguous support to them. We also proposed PS3-RAM, which is a fast, portable and scalable statistical STT-RAM reliability analysis model [Wen 2012]. Design Space Exploration and Optimization With the support of these models, we explore different device/circuit optimization techniques. For example, in [Niu 2012a] we studied the power reduction technique for the application of ECC scheme in ReRAM designs and proposed to use ECC code to relax the BER (Bit Error Rate) requirement of a single memory to improve the write energy consumption and latency for both 1T1R and cross-point ReRAM designs. In [Xu 2011], we proposed a methodology to design STT-RAM for different optimization goals such as read performance, write performance and write energy by leveraging the trade-off between write current and write time of MTJ. We also studied the tradeoffs in building a reliable crosspoint ReRAM array [Niu 2012b]. We have conducted an in depth analysis of the circuit and system level design implications of multi-layer cross-point Resistive RAM (MLCReRAM) from performance, power and reliability perspectives [Xu 2013]. The objective of this study is to understand the design trade-offs of this technology with respect to the MLC Phase Change Memory (MLCPCM).Our MLC ReRAM design at the circuit and system levels indicates that different resistance allocation schemes, programming strategies, peripheral designs, and material selections profoundly affect the area, latency, power, and reliability of MLC ReRAM. Based on this analysis, we conduct two case studies: first we compare MLC ReRAM design against MLC phase-change memory (PCM) and multi-layer cross-point ReRAM design, and point out why multi-level ReRAM is appealing; second we further explore the design space for MLC ReRAM. Architecture and Application We explored hybrid checkpointing using phase-change memory for future exascale systems [Dong 2011] and showed that the use of nonvolatile memory for local checkpointing significantly increases the number of faults covered by local checkpoints and reduces the probability of a global failure in the middle of a global checkpoint to less than 1%. We also proposed a technique called i2WAP to mitigate the write variations in NVM-based last-level cache for the improvement of the NVM lifetime [Wang 2013]. Our wear leveling technique attempts to work around the limitations of write endurance by arranging data access so that write operations can be distributed evenly across all the storage cells. During our intensive research on fault-tolerant NVM design, we found that ECC cannot effectively tolerate hard errors from limited write endurance and process imperfection. Therefore, we devised a novel Point and Discard (PAD) architecture in in [ 2012] as a hard-error-tolerant architecture for ReRAM-based Last Level Caches. PAD improves the lifetime of ReRAM caches by 1.6X-440X under different process variations without performance overhead in the system's early life. We have investigated the applicability of NVM for persistent memory design [Zhao 2013]. New byte addressable NVM enables fast persistent memory that allows in-memory persistent data objects to be updated with much higher throughput. Despite the significant improvement, the performance of these designs is only 50% of the native system with no persistence support, due to the logging or copy-on-write mechanisms used to update the persistent memory. A challenge in this approach is therefore how to efficiently enable atomic, consistent, and durable updates to ensure data persistence that survives application and/or system failures. We have designed a persistent memory system, called Klin, that can provide performance as close as that of the native system. The Klin design adopts a non-volatile cache and a non-volatile main memory for constructing a multi-versioned durable memory system, enabling atomic updates without logging or copy-on-write. Our evaluation shows that the proposed Kiln mechanism can achieve up to 2X of performance improvement to NVRAM-based persistent memory employing write-ahead logging. In addition, our design has numerous practical advantages: a simple and intuitive abstract interface, microarchitecture-level optimizations, fast recovery from failures, and no redundant writes to slow non-volatile storage media. The work was published in MICRO 2013 and received Best Paper Honorable Mentioned Award.
Future exascale systems must be optimized for both power and performance at scale in order to achieve DOE’s goal of a sustained petaflop within 20 Megawatts by 2022 [1]. Massive parallelism of the future systems combined with complex memory hierarchies will form a barrier to efficient application and architecture design. These challenges are exacerbated with emerging complex architectures such as GPGPUs and Intel Xeon Phi as parallelism increases orders of magnitude and system power consumption can easily triple or quadruple. Therefore, we need techniques that can reduce the search space for optimization, isolate power-performance bottlenecks, identify root causes for software/hardware inefficiency, and effectively direct runtime scheduling.