New SOA treatments within the Energy Exascale Earth System Model (E3SM): Strong production and sinks govern atmospheric SOA distributions and radiative forcing
Explore the source record for details and available documents.
Lou, Sijia↗
SEARCH · Engineering Papers
Results for “EXASCALE”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Breaking the System Noise Barrier at Exascale
Explore the source record for details and available documents.
Computer science↗
Program Development Tools and Infrastructures: Activities by ASC CSSE/CCE and Partners
Exascale class machines will exhibit a new level of complexity: they will feature an unprecedented number of cores and threads, will most likely be heterogeneous and deeply hierarchical, and offer a range of new hardware techniques (such as speculative threading, transactional memory, programmable prefetching, and programmable accelerators), which all have to be utilized for an application to realize the full potential of the machine. Additionally, users will be faced with less memory per core, fixed total power budgets, and sharply reduced MTBFs. At the same time, it is expected that the complexity of applications will rise sharply for exascale systems, both to implement new science possible at exascale and to exploit the new hardware features necessary to achieve exascale performance. This is particularly true for many of the NNSA codes, which are large and often highly complex integrated simulation codes that push the limits of everything in the system including language features. To overcome these limitations and to enable users to reach exascale performance, users will expect a new generation of tools that address the bottlenecks of exascale machines, that work seamlessly with the (set of) programming models on the target machines, that scale with the machine, that provide automatic analysis capabilities, and that are flexible and modular enough to overcome the complexities and changing demands of the exascale architectures. Further, any tool must be robust enough to handle the complexity of large integrated codes while keeping the user's learning curve low. With the ASC program, in particular the CSSE (Computational Systems and Software Engineering) and CCE (Common Compute Environment) projects, we are working towards a new generation of tools that fulfill these requirements and that provide our users as well as the larger HPC community with the necessary tools, techniques, and methodologies required to make exascale performance a reality.
43 PARTICLE ACCELERATORS↗
ECP libraries and tools: An overview
The Exascale Computing Project (ECP) Software Technology and Co-Design teams addressed the growing complexities in high-performance computing (HPC) by developing scalable software libraries and tools that leverage exascale system capabilities. As we enter the exascale era, the need for reusable, optimized software solutions that can handle the unique challenges posed by these systems becomes increasingly important. The primary challenges the ECP teams faced were to create software libraries and tools that are performant on exascale architectures and portable and usable across diverse hardware platforms. Efforts addressed issues related to concurrent execution, memory management, and the integration of heterogeneous computing resources, such as GPUs from multiple vendors. The ECP’s strategy involved a structured development process encompassing the creation, optimization, and deployment of software in collaboration with industry, academia, and national laboratories. The project was organized into several technical areas: co-design of domain-specific suites with target applications, programming models and runtimes, development tools, mathematical libraries, data and visualization tools, and software ecosystem and delivery mechanisms. ECP has successfully developed a large portfolio of software libraries and tools that demonstrate significant improvements in performance and scalability on exascale systems. These products have been integrated into the Department of Energy’s computing facilities, supporting various scientific applications and ensuring robust performance across different hardware setups. ECP advancements in software development for exascale computing highlight the importance of a collaborative and adaptive approach to handling next-generation HPC systems complexities. The lessons learned emphasize the need for continuous engagement with end-users and vendors, and the importance of maintaining a balance between innovation and practical implementation. Future efforts will focus on ensuring scalability, keeping pace with rapid hardware advancements, and further enhancing the interoperability and usability of the software ecosystem. In conclusion, subsequent articles in this special issue provide in-depth discussions and case studies into specific library and tool efforts.
97 MATHEMATICS AND COMPUTING↗
Advanced Turbulence Models for Large-Scale Atmospheric Boundary Layer Flows
We present high-fidelity large-eddy-simulation (LES) modeling approaches for the turbulent atmospheric boundary layer (ABL) flows. Wind energy is a prime example of an application driven by ABL. Generation of electrical energy from farms of wind turbines at night in the stable ABL is a particularly interesting situation. In this report, we consider the well-known GEWEX (Global Energy and Water Cycle Experiment) Atmospheric Boundary Layer Study (GABLS) stably stratified benchmark LES case. We use a high-order spectral element code Nek5000/RS, which is supported under the DOE's Exascale Computing Project (ECP) Center for Efficient Exascale Discretizations (CEED) project, targeting application simulations on various acceleration-device based exascale computing platforms. In our earlier ANL report, we demonstrated our newly developed subgrid-scale (SGS) models based on high-pass filter (HPF), mean-field eddy viscosity (MFEV), and Smagorinsky (SMG) with no-slip and traction boundary conditions, provided with low-order statistics, convergence and turbulent structure analysis. In this report, we extend the range of our SGS modeling approaches in the context of the mean-field eddy viscosity (MFEV), to include the solution of an SGS turbulent kinetic energy equation (TKE). We demonstrate the model fidelity of Nek5000/RS in comparison to that of AMR-Wind, a block-structured second-order finite-volume code with adaptive-mesh-refinement capabilities, with which we studied scaling performance for both codes in comparison on DOE's leadership computing platforms.
17 WIND ENERGY↗
FOX: A Fault Oblivious Extreme-Scale Execution Environment
The FOX project, funded under the ASCR X-stack I program, developed systems software and runtime libraries for a new approach to the data and work distribution for massively parallel, fault oblivious application execution. Our work was motivated by the premise that exascale computing systems will provide a thousand-fold increase in parallelism and a proportional increase in failure rate relative to today’s machines. To deliver the capability of exascale hardware, the systems software must provide the infrastructure to support existing applications while simultaneously enabling efficient execution of new programming models that naturally express dynamic, adaptive, irregular computation; coupled simulations; and massive data analysis in a highly unreliable hardware environment with billions of threads of execution. Our OS research has prototyped new methods to provide efficient resource sharing, synchronization, and protection in a many-core compute node. We have experimented with alternative task/dataflow programming models and shown scalability in some cases to hundreds of thousands of cores. Much of our software is in active development through open source projects. Concepts from FOX are being pursued in next generation exascale operating systems. Our OS work focused on adaptive, application tailored OS services optimized for multi → many core processors. We developed a new operating system NIX that supports role-based allocation of cores to processes which was released to open source. We contributed to the IBM FusedOS project, which promoted the concept of latency-optimized and throughput-optimized cores. We built a task queue library based on distributed, fault tolerant key-value store and identified scaling issues. A second fault tolerant task parallel library was developed, based on the Linda tuple space model, that used low level interconnect primitives for optimized communication. We designed fault tolerance mechanisms for task parallel computations employing work stealing for load balancing that scaled to the largest existing supercomputers. Finally, we implemented the Elastic Building Blocks runtime, a library to manage object-oriented distributed software components. To support the research, we won two INCITE awards for time on Intrepid (BG/P) and Mira (BG/Q). Much of our work has had impact in the OS and runtime community through the ASCR Exascale OS/R workshop and report, leading to the research agenda of the Exascale OS/R program. Our project was, however, also affected by attrition of multiple PIs. While the PIs continued to participate and offer guidance as time permitted, losing these key individuals was unfortunate both for the project and for the DOE HPC community.
97 MATHEMATICS AND COMPUTING↗
2014 Runtime Systems Summit. Runtime Systems Report
This report summarizes runtime system challenges for exascale computing, that follow from the fundamental challenges for exascale systems that have been well studied in past reports, e.g., [6, 33, 34, 32, 24]. Some of the key exascale challenges that pertain to runtime systems include parallelism, energy efficiency, memory hierarchies, data movement, heterogeneous processors and memories, resilience, performance variability, dynamic resource allocation, performance portability, and interoperability with legacy code. In addition to summarizing these challenges, the report also outlines different approaches to addressing these significant challenges that have been pursued by research projects in the DOE-sponsored X-Stack and OS/R programs. Since there is often confusion as to what exactly the term “runtime system” refers to in the software stack, we include a section on taxonomy to clarify the terminology used by participants in these research projects. In addition, we include a section on deployment opportunities for vendors and government labs to build on the research results from these projects. Finally, this report is also intended to provide a framework for discussing future research and development investments for exascale runtime systems, and for clarifying the role of runtime systems in exascale software.
97 MATHEMATICS AND COMPUTING↗
ECP Software Technology Capability Assessment Report
The Exascale Computing Project Software Technology (ECP ST) focus area represents the key bridge between Exascale systems and the scientists developing applications that will run on those platforms. ECP offers a unique opportunity to build a coherent set of software (often referred to as the "software stack") that will allow application developers to maximize their ability to write highly parallel applications, targeting multiple Exascale architectures with runtime environments that will provide high performance and resilience. But applications are only useful if they can provide scientific insight, and the unprecedented data produced by these applications require a complete analysis work ow that includes new technology to scalably collect, reduce, organize, curate, and analyze the data into actionable decisions. This requires approaching scientific computing in a holistic manner, encompassing the entire user workflow - from conception of a problem, setting up the problem with validated inputs, performing high-fidelity simulations, to the application of uncertainty quantification to the final analysis. The software stack plan defined here aims to address all of these needs by extending current technologies to Exascale where possible, by performing the research required to conceive of new approaches necessary to address unique problems where current approaches will not suffice, and by deploying high-quality and robust software products on the platforms developed in the Exascale systems project. The ECP ST portfolio has established a set of interdependent projects that will allow for the research, development, and delivery of a comprehensive software stack,
97 MATHEMATICS AND COMPUTING↗
Non-Proprietary Companion to the Q1 CY2021 PathForward Final Assessment (WBS 2.4.1, Milestone PM-HI-1040)
The PathForward element of the Exascale Computing Project (ECP) Hardware and Integration (HI) focus area has met its goals of preparing the US industry for exascale system procurements and generally improving US competitiveness in the worldwide computing market. This report is a companion to the deliverable for the ECP milestone PM-HI-1040, Assess PathForward Impact Against Exascale Hardware Challenges, which gives a summary of the final status of each PathForward project, describes progress achieved against PathForward contract milestones, and includes a final assessment of each vendor’s progress on key exascale challenges. This companion document details the results of the PathForward research to the extent possible without disclosing proprietary information and the impact on products and US exascale systems. In addition, it captures lessons learned in order to inform future projects in general and in high-performance computing in particular.
97 MATHEMATICS AND COMPUTING↗
ACDT: Architected Composite Data Types Trading-in Unfettered Data Access for Improved Execution
With Exascale performance and its challenges in mind, one ubiquitous concern among architects is energy efficiency. Petascale systems projected to Exascale systems are unsustainable at current power consumption rates. One major contributor to system-wide power consumption is the number of memory operations leading to data movement and management techniques applied by the runtime system. To address this problem, we present the concept of the Architected Composite Data Types (ACDT) framework. The framework is made aware of data composites, assigning them a specific layout, transformations and operators. Data manipulation overhead is amortized over a larger number of elements and program performance and power efficiency can be significantly improved. We developed the fundamentals of an ACDT framework on a massively multithreaded adaptive runtime system geared towards Exascale clusters. Showcasing the capability of ACDT, we exercised the framework with two representative processing kernels - Matrix Vector Multiply and the Cholesky Decomposition – applied to sparse matrices. As transformation modules, we applied optimized compress/decompress engines and configured invariant operators for maximum energy/performance efficiency. Additionally, we explored two different approaches based on transformation opaqueness in relation to the application. Under the first approach, the application is agnostic to compression and decompression activity. Such approach entails minimal changes to the original application code, but leaves out potential applicationspecific optimizations. The second approach exposes the decompression process to the application, hereby exposing optimization opportunities that can only be exploited with application knowledge. The experimental results show that the two approaches have their strengths in HW and SW respectively, where the SW approach can yield performance and power improvements that are an order of magnitude better than ACDT-oblivious, hand-optimized implementations.We consider the ACDT runtime framework an important component of compute nodes that will lead towards power efficient Exascale clusters.
exascale↗
From PeleC to PeleACC, to PeleC++
PeleC is an Exascale Computing Project application for simulating compressible combustion in complex geometries. It has been built on top of the popular AMReX library. In the beginning of the Exascale Computing Project, PeleC was focused on KNL. It uses a mixture of C++, C, and kernels written in Fortran to obtain performance by focusing on vectorization. Recently we have taken two approaches in deciding PeleC's future for obtaining performance on exascale GPU machines. In the first programming model, we decorated the Fortran kernels with OpenACC directives. This expedited our ability to run at large scales on Summit's GPUs, where we achieved a significant speedup over the CPUs on Summit. The second programming model involved rewriting the Fortran kernels in C++ and using AMReX's Kokkos-like lambda abstractions for running on the GPU. This resulted in similar speedups on Summit's GPUs over merely utilizing the CPUs. Both approaches involved AMReX's management of memory transfers between the device and host. In this work, we compare and contrast the benefits and pitfalls to both programming approaches regarding performance, performance portability, and productivity. We also discuss advantages we have found in taking the time to modernize our code and why have chosen a specific pathway to prepare our code for the future DOE exascale machines.
exascale computing↗
High-Performance Computing for Earth System Modeling
High-performance computing (HPC) plays an important role during the development of Earth system models. This chapter reviews HPC efforts related to Earth system models, including community Earth system models and energy exascale Earth system models. Specifically, this chapter evaluates computational and software design issues, analyzes several current HPC-related model developments, and provides an outlook for some promising areas within Earth system modeling in the era of exascale computing.
Wang, Dali↗
Evaluating Online Global Recovery with Fenix Using Application-Aware In-Memory Checkpointing Techniques
Exascale systems promise the potential for computation at unprecedented scales and resolutions, but achieving exascale by the end of this decade presents significant challenges. A key challenge is due to the very large number of cores and components and the resulting mean time between failures (MTBF) in the order of hours or minutes. Since the typical run times of target scientific applications are longer than this MTBF, fault tolerance techniques will be essential. An important class of failures that must be addressed is process or node failures. While checkpoint/restart (C/R) is currently the most widely accepted technique for addressing processor failures, coordinated, stable-storage-based global C/R might be unfeasible at exascale when the time to checkpoint exceeds the expected MTBF. This paper explores transparent recovery via implicitly coordinated, diskless, application-driven checkpointing as a way to tolerate process failures in MPI applications at exascale. The discussed approach leverages User Level Failure Mitigation (ULFM), which is being proposed as an MPI extension to allow applications to create policies for tolerating process failures. Specifically, this paper demonstrates how different implementations of application-driven in-memory checkpoint storage and recovery compare in terms of performance and scalability. We also experimentally evaluate the effectiveness and scalability of the Fenix online global recovery framework on a production system -the Titan Cray XK7 at ORNL- and demonstrate the ability of Fenix to tolerate dynamically injected failures using the execution of four benchmarks and mini-applications with different behaviors.
97 MATHEMATICS AND COMPUTING↗
HPDR: High-Performance Portable Scientific Data Reduction Framework
The rapid growth in scientific data generation is outpacing advancements in computing systems necessary for efficient storage, transfer, and analysis, particularly in the context of exascale computing. With the deployment of first-generation exascale computing systems and next-generation experimental facilities, this gap is widening and necessitates effective data reduction techniques to manage enormous data volumes. Over the past decade, various data reduction methods, including lossless compression, error-controlled lossy compression, and data refactoring, have been developed to accelerate I/O in scientific workflows. Despite significant reductions in data volume, these methods introduce considerable computational overhead, which can become the new bottleneck in data processing. To mitigate this, GPU-accelerated data reduction algorithms have been introduced. However, challenges remain in their integration into exascale workflows, including limited portability across different GPU architectures, substantial memory transfer overhead, and reduced scalability on dense multi-GPU systems. To address these challenges, we propose HPDR, a high-performance and portable data reduction framework. HPDR is designed to enable the execution of state-of-the-art reduction algorithms across diverse processor architectures while reducing memory transfer overhead to 2.3 % of the original, resulting in up to 3.5× faster throughput compared to existing solutions. It also achieves up to 96% of the theoretical speedup in multi-GPU settings. In addition, evaluations on accelerating I/O operations at scale up to 1,024 nodes of the Frontier supercomputer demonstrate that HPDR can achieve up to 103 TB/s reduction throughput, providing up to 4× acceleration in parallel I/O performance compared to existing data reduction routines. This work highlights the potential of HPDR to significantly enhance data reduction efficiency in exascale computing environments.
Chen, Jieyang [University of Oregon]↗
An Evaluation of the Effect of Network Cost Optimization for Leadership Class Supercomputers
Dragonfly-based networks are an extensively deployed network topology in large-scale high-performance computing due to their cost-effectiveness and efficiency. The US will soon have three Exascale supercomputers for leadership class workloads deployed using dragonfly networks. Compared to indirect networks of similar scale, the dragonfly network has considerably reduced cable lengths, cable counts, and switch counts, resulting in significant network cost savings for a given system size, however, these cost reductions result in reduced global minimal paths and more challenging routing. Additionally, large scale dragonfly networks often require a taper at the global link level, resulting in less bisection bandwidth than is achievable in other traditional non-blocking topologies of equivalent scale. While dragonfly networks have been extensively studied, they have yet to be fully evaluated in an extreme scale (i.e., exascale) system that targets capability workloads. In this paper, we present the results of the first large scale evaluation of a dragonfly network on an exascale system (Frontier) and compare its behavior to a similar scale fat-tree network on a previous generation TOP500 system (Summit). This evaluation aims to determine the effect of network cost optimizations by measuring a tapered topology’s impact on capability workloads. Our evaluation is based on a collection of synthetic microbenchmarks, mini-apps, and full scale applications. It compares the scaling efficiencies of each benchmark between the dragonfly-based Frontier and the fat-tree-based Summit systems. Our results show that a dragonfly network is $\sim \mathbf{3 0 \%}$ more cost efficient than a fat-tree topology, which amortizes to $\sim 3 \%$ of an exascale system cost. Furthermore, while tapered dragonfly networks impose significant tradeoffs, the impacts are not as broad as initially thought and are mostly seen in applications with global communication patterns, particularly all-to-all (e.g., FFT-based algorithms), but also local communication patterns (e.g., nearest-neighbor algorithms) that are sensitive to network performance variability.
Khan, Awais↗
A performance model for the communication in fast multipole methods on high-performance computing platforms
Exascale systems are predicted to have approximately 1 billion cores, assuming gigahertz cores. Limitations on affordable network topologies for distributed memory systems of such massive scale bring new challenges to the currently dominant parallel programing model. Currently, there are many efforts to evaluate the hardware and software bottlenecks of exascale designs. It is therefore of interest to model application performance and to understand what changes need to be made to ensure extrapolated scalability. The fast multipole method (FMM) was originally developed for accelerating N-body problems in astrophysics and molecular dynamics but has recently been extended to a wider range of problems. Its high arithmetic intensity combined with its linear complexity and asynchronous communication patterns make it a promising algorithm for exascale systems. In this paper, we discuss the challenges for FMM on current parallel computers and future exascale architectures, with a focus on internode communication. We focus on the communication part only; the efficiency of the computational kernels are beyond the scope of the present study. We develop a performance model that considers the communication patterns of the FMM and observe a good match between our model and the actual communication time on four high-performance computing (HPC) systems, when latency, bandwidth, network topology, and multicore penalties are all taken into account. To our knowledge, this is the first formal characterization of internode communication in FMM that validates the model against actual measurements of communication time. The ultimate communication model is predictive in an absolute sense; however, on complex systems, this objective is often out of reach or of a difficulty out of proportion to its benefit when there exists a simpler model that is inexpensive and sufficient to guide coding decisions leading to improved scaling. The current model provides such guidance.
Computer Science↗
NGEE Arctic Integrated Modeling (IM2): Improved subgrid hillslope hydrologic connectivity
This data product represents the integration of new code capability for arctic tundra hillslope hydrologic processes into the Energy Exascale Earth System Model (E3SM), through the E3SM Land Model (ELM) component. This code integration is the result of collaborative effort between the NGEE Arctic project and the E3SM project. The current ELM represents water movement primarily through vertical processes, such as precipitation, canopy interception, evaporation, infiltration, and soil water movement. Lateral water movement—such as surface runoff, subsurface flow, and river transport—plays a significant role in the hydrological cycle, especially in regions with varied topography. While E3SM includes a runoff routing component representing water transport in the river network, the lateral transport of water at the subgrid scale within the land model has previously not been taken into account. With the recent development of topographic units within the ELM subgrid data structure, there is an opportunity to simulate hillslope hydrologic connectivity by introducing water transport along topographic gradients. We expect that more realistic representation of hillslope hydrologic processes will lead to improved predictions of both soil water content and river network flows. Lateral transport of water at and near the surface is represented as a sub-grid process in this new code development. Water is tracked as it moves from higher to lower elevations within a gridcell. This capability uses the nested hierarchical sub-grid scheme within ELM to connect water fluxes from sub-grid elements with higher elevation to those with lower elevation. This data record consists of a single document (pdf format) that describes the theoretical basis for the hillslope hydrology processes added to ELM, and describes the modifications made to the ELM code. The Methods section of this metadata record includes a link to the public E3SM code repository where the exact code modifications as integrated in E3SM can be accessed. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).
Thornton, Peter E [ORNL] (ORCID:0000000247595158)↗
NGEE Arctic Integrated Modeling (IM3): Improved snow-vegetation interaction
This data product represents the integration of new code capability for arctic tundra snow-vegetation-terrain interactions into the Energy Exascale Earth System Model (E3SM), through the E3SM Land Model (ELM) component. This code integration is the result of collaborative effort between the NGEE Arctic project and the E3SM project. The NGEE Arctic project developed a total of six Integrated Modeling (IM) modules informed by observations and experiments. New ELM capability represented by this data product (IM3) falls into three categories: 1) Downscaling from gridcell to topographic unit level when working through the existing coupler bypass code. 2) Four new parameters (taper, stocking, bendresist, and vegshape) have been added to ELM to allow for flexible definition of snow-vegetation interactions. 3) Vegshape and bendresist parameters are used to calculate the fraction of leaf area and/or stem area buried by snow for a given snow depth. This data record consists of a single document (pdf format) that describes the theoretical basis for the snow-vegetation-terrain interactions added to ELM, and describes the modifications made to the ELM code. The Methods section of this metadata record includes a link to the public E3SM code repository where the exact code modifications as integrated in E3SM can be accessed. The Next-Generation Ecosystem Experiments: Arctic (NGEE Arctic), was a research effort to reduce uncertainty in Earth System Models by developing a predictive understanding of carbon-rich Arctic ecosystems and feedbacks to climate. NGEE Arctic was supported by the Department of Energy's Office of Biological and Environmental Research. The NGEE Arctic project had two field research sites: 1) located within the Arctic polygonal tundra coastal region on the Barrow Environmental Observatory (BEO) and the North Slope near Utqiagvik (Barrow), Alaska and 2) multiple areas on the discontinuous permafrost region of the Seward Peninsula north of Nome, Alaska. Through observations, experiments, and synthesis with existing datasets, NGEE Arctic provided an enhanced knowledge base for multi-scale modeling and contributed to improved process representation at global pan-Arctic scales within the Department of Energy's Earth system Model (the Energy Exascale Earth System Model, or E3SM), and specifically within the E3SM Land Model component (ELM).
Thornton, Peter E [ORNL] (ORCID:0000000247595158)↗