Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “EXASCALE”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 973 records · Page 54
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

Secure Collaborative Environment for Seamless Sharing of Scientific Knowledge

In a secure collaborative environment, tera-bytes of data generated from powerful scientific instruments are used to train secure machine learning (ML) models on exascale computing systems, which are then securely shared with internal or external collaborators as cloud-based services. Devising such a secure platform is necessary for seamless scientific knowledge sharing without compromising individual, or institute-level, intellectual property and privacy details. By enabling new computing opportunities with sensitive data, we envision a secure collaborative environment that will play a significant role in accelerating scientific discovery. Several recent technological advancements have made it possible to realize these capabilities. In this paper, we present our efforts at ORNL toward developing a secure computation platform. We present a use case where scientific data generated from complex instruments, like those at the Spallation Neutron Source (SNS), are used to train a differential privacy enabled deep learning (DL) network on Summit, which is then hosted as a secure multi-party computation (MPC) service on ORNL’s Compute and Data Environment for Science (CADES) cloud computing platform for third-party inference. In this feasibility study, we discuss the challenges involved, elaborate on leveraged technologies, analyze relevant performance results and present the future vision of our work to establish secure collaboration capabilities within and outside of ORNL.

Yoginath, Srikanth↗
DOE OSTI2023

Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems

Designing and refactoring complex scientific code, such as the E3SM land model (ELM), for new computing architectures is challenging. This paper presents design strategies and technical approaches to develop a data-oriented, GPU-ready ELM model using compiler directives (OpenACC/OpenMP). We first analyze the datatypes and processes in the original ELM code. Then we present design considerations for ultrahigh-resolution ELM (uELM) development for massive GPU systems. These techniques include the global data-oriented simulation workflow, domain partition, code porting and data copy, memory reduction, parallel loop restructure and flattening, and race condition detection. We implemented the first version of uELM using OpenACC targeting the NVidia GPUs in the Summit supercomputer at Oak Ridge National Laboratory. During the implementation, we developed a software tool (named SPEL) to facilitate code generation, verification, and performance tuning using these techniques. The first uELM implementation for Nvidia GPUs on Summit delivered promising results: 1) over 98% of the ELM code was automatically generated and tuned by scripts. Most ELM modules had better computational performances than the original ELM code for CPUs. The GPU-ready uELM is more scalable than the CPU code on fully-loaded Summit nodes. Example profiling results from several modules are also presented to illustrate the performance improvements and race condition detection. The lessons learned and toolkit developed in the study are also suitable for further uELM deployment using OpenMP on the first US exascale computer, Frontier, equipped with AMD CPUs and GPUs.

Schwartz, Peter↗
DOE OSTI2024

CI/CD Efforts for Validation, Verification and Benchmarking OpenMP Implementations

Software developers must adapt to keep up with the changing capabilities of platforms so that they can utilize the power of High-Performance Computers (HPC), including exascale systems. OpenMP, a directive-based parallel programming model, allows developers to include directives to existing C, C++, or Fortran code to allow node level parallelism without compromising performance. This paper describes our CI/CD efforts to provide easy evaluation of the support of OpenMP across different compilers using existing testsuites and benchmark suites on HPC platforms. Our main contributions include (1) the set of a Continuous Integration (CI) and Continuous Development (CD) workflow that captures bugs and provides faster feedback to compiler developers, (2) an evaluation of OpenMP (offloading) implementations supported by AMD, HPE, GNU, LLVM, and Intel, and (3) evaluation of the quality of compilers across different heterogeneous HPC platforms. With the comprehensive testing through the CI/CD workflow, we aim to provide a comprehensive understanding of the current state of OpenMP (offloading) support in different compilers and heterogeneous platforms consisting of CPUs and GPUs from NVIDIA, AMD, and Intel.

Jarmusch, Aaron↗
DOE OSTI2025

Flexible User-Defined Domain Decomposition in Kilometer-Scale E3SM Land Model Simulation

The Energy Exascale Earth System Model (E3SM) Land Model (ELM) has been extended to kilometer-scale (km-ELM) resolutions, enabling high-fidelity simulations of terrestrial processes at 1 km x 1 km grid spacing. In ELM, domain decomposition partitions the computational domain across processors, ensuring efficient parallel execution. Currently, round-robin decomposition is applied, providing a straightforward way to distribute computational workload. As ELM continues evolving at the kilometer-scale (km-scale), particularly with integrating lateral flow modeling, decomposition strategies must also account for the increased workload and data movement. This paper introduces a flexible user-defined domain decomposition framework, allowing users to customize domain partitioning based on application requirements. The impact of different decomposition strategies is evaluated across various applications concerning computation, communication, and I/O. Results demonstrate that while 1D partitioning yields superior I/O performance, k-nearest neighbors (KNN) clustering effectively reduces inter-process communication overhead. This study lays the groundwork for scalable partitioning in large-scale land surface simulations, enhancing next-generation Earth system modeling.

Wang, Dali [ORNL] (ORCID:0000000168065108)↗
DOE OSTI2017

Optimizing fusion PIC code performance at scale on Cori Phase 2

In this paper we present the results of optimizing the performance of the gyrokinetic full-f fusion PIC code XGC1 on the Cori Phase Two Knights Landing system. The code has undergone substantial development to enable the use of vector instructions in its most expensive kernels within the NERSC Exascale Science Applications Program. We study the single-node performance of the code on an absolute scale using the roofline methodology to guide optimization efforts. We have obtained 2x speedups in single node performance due to enabling vectorization and performing memory layout optimizations. On multiple nodes, the code is shown to scale well up to 4000 nodes, near half the size of the machine. We discuss some communication bottlenecks that were identified and resolved during the work.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2018

Performance and Energy Usage of Workloads on KNL and Haswell Architectures. In: High Performance Computing Systems. Performance Modeling, Benchmarking, and Simulation

Manycore architectures are an energy-efficient step towards exascale computing within a constrained power budget. The Intel Knights Landing (KNL) manycore chip is a specific example of this and has seen early adoption by a number of HPC facilities. It is therefore important to understand the performance and energy usage characteristics of KNL. In this paper, we evaluate the performance and energy efficiency of KNL in contrast to the Xeon (Haswell) architecture for applications representative of the workload of users at NERSC. We consider the optimal MPI/OpenMP configuration of each application and use the results to characterize KNL in contrast to Haswell. As well as traditional DDR memory, KNL contains MCDRAM and we also evaluate its efficacy. Our results show that, averaged over our benchmarks, KNL is 1.84 × more energy efficient than Haswell and has 1.27 × greater performance.

Benchmarking↗
DOE OSTI2018

SharP Unified Memory Allocator: An Intent-Based Memory Allocator for Extreme-Scale Systems

The pre-exascale systems will soon be deployed with a deep, complex memory hierarchy composed of many heterogeneous memories. This presents multiple challenges for users including: how to allocate data objects with locality between memories and devices for the various memories in these systems, which includes DRAM, High-bandwidth Memory (HBM), and non-volatile random access memory (NVRAM), and how to perform these allocations while providing portability for their application. Currently, the user can make use of multiple, disjoint libraries to allocate data objects on these memories. However, it is difficult to obtain locality between memories and devices when using libraries that are unaware of each other. This paper presents the Unified Memory Allocator (UMA) of the SHARed data-structure centric Programming abstraction (SharP) library, which provides a unified interface for memory allocations across DRAM, HBM, and NVRAM and is extensible to support future memory types. In addition, the SharP UMA allows for portability between systems by supporting both explicit and implicit, intent-based memory allocations. To demonstrate the ease of use of the SharP UMA, we have extended both Open MPIand OpenSHMEM-Xto support SharP. We validate this work by evaluating the performance implications and intent-based approach with synthetic benchmarks as well as adaptations of the Graph500 benchmark.

Aderholdt, William↗
DOE OSTI2018

OpenMP 4.5 Validation and Verification Suite for Device Offload

OpenMP has been widely adopted for shared memory systems for over a decade. With the heterogeneity trend in architectures rapidly growing, the programming model needed to evolve such that applications could not only be ported to traditional CPUs but also to accelerators often acting as discrete or integrated devices to CPUs. To that end, OpenMP started to provide support for heterogeneous systems since 2013 when the version 4.0 of the specification was ratified. OpenMP 4.5 is being enhanced to cover major requirements of Exascale Computing Project (ECP) applications. As a result it is time-critical to ensure that the implementations of the 4.5 features are correct and conforming to the specification. This paper focuses on building a Validation and Verification testsuite that will test and present results for several offloading features implemented in compilers such as Clang, IBM XL C/C++, CCE, and GCC. We have results for our testsuite on TITAN, Summitdev and Summit at the Oak Ridge National Lab. We will highlight some of the ambiguities we encountered in the process of validating and verifying feature implementations. We also make the testsuite available for anyone to use and will walk the readers through the infrastructure and the workflow of the testsuite. A website has been built to capture our efforts narrated in this paper https://crpl.cis.udel.edu/ompvvsollve.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2012

DAG-Based Software Frameworks for PDEs

The task-based approach to software and parallelism is well-known and has been proposed as a potential candidate, named the silver model, for exascale software. This approach is not yet widely used in the large-scale multi-core parallel computing of complex systems of partial differential equations. After surveying task-based approaches we investigate how well the Uintah software and an extension named Wasatch fit in the task-based paradigm and how well they perform on large scale parallel computers. The conclusion is that these approaches show great promise for petascale but that considerable algorithmic challenges remain.

Directed Acyclic Graph↗
DOE OSTI2013

A Similarity-Based Analysis Tool for Porting OpenMP Applications. In: Facing the Multicore-Challenge III. Lecture Notes in Computer Science, vol 7686.

Exascale computers are expected to exhibit an unprecedented level of complexity, thereby posing significant challenges for porting applications to these new systems. One of the ways to support this transition is to create tools that allow their users to benefit from prior successful porting experiences. The key to such an approach is the manner in which we define source code similarity, and whether similar codes can be ported in the same way to a given system. In this paper, we propose a novel approach based on the notion of similarity that uses static and dynamic code features to check if two serial subroutines can be ported with the same OpenMP strategy. Our approach creates an annotated family distance tree based on the syntactic structure of subroutines, where subroutines that belong to the same syntactic family and share the similar code features have a greater potential to be optimized in the same way. We describe the design and implementation of a tool, based upon a compiler and performance tool, that is used to gather the data to build this porting planning tree. We then validate our approach by analyzing the similarity in subroutines of the serial version of the NAS benchmarks and comparing how they were ported in the OpenMP version of the suite.

Ding, Wei↗
DOE OSTI2020

Wintertime Arctic Oscillation and North Atlantic Oscillation and their impacts on the Northern Hemisphere climate in E3SM

The characteristics of the wintertime Arctic Oscillation (AO) and North Atlantic Oscillation (NAO) and their impacts on climate variability over the Northern Hemisphere are important metrics for evaluating a climate system model. Observational analyses reveal that the horizontal and vertical structures in the AO and NAO exhibit a meridional dipole and a large-scale barotropic pattern between the Arctic and mid-latitudes. Historical model simulations from the Energy Exascale Earth System Model (E3SM-HIST) are used to identify how well it captures these major climate modes. It is found that the simulated AO and NAO modes have spatial structures similar to the observed features. In addition, the observed frequency bands in the AO and NAO-related time variability are captured well in the E3SM-HIST simulation. Associated with the positive phase in wintertime AO and NAO, zonal flow and warm advection in mid-latitude continents are enhanced, along with stronger cold flow from enhanced northerly winds over high latitudes. These features are linked to the atmospheric circulation pattern reflected by lower SLP anomalies over the Arctic and higher SLP anomalies over the mid-latitudes. In E3SM-HIST, these spatial associations and main structural features are analogous to those in observations. In the time-height evolution related to winter AO and NAO modes, it can also be seen that the simulations reproduce the downward propagating patterns in observations. Nevertheless, the vertical structures associated with AO and NAO in E3SM-HIST exhibit substantial biases in the lower stratosphere. The cause of these stratospheric biases is investigated using the strength of climatological stratospheric polar vortex (SPV) and wave activity fluxes. The results herein suggest that E3SM-HIST has a reasonable skill in reproducing the observed characteristics related to the winter AO and NAO, although there exist systematic biases in the associated climate variability.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2023

Region and cloud regime dependence of parametric sensitivity in E3SM atmosphere model

Abstract The Department of Energy (DOE)’s Energy Exascale Earth System Model (E3SM), including its atmosphere model (EAM), has many relatively new features. In a previous study we conducted a systematic parametric sensitivity analysis for EAM based on short, perturbed parameter ensemble (PPE) simulations, mainly focusing on global mean climate features and metrics. While parameter values in global climate models are generally invariant in space and time, model response to parameters perturbation may vary by regions and climate regimes, which motivates the need to better understand the EAM model behaviors and physics at regional scale and process level. In this study, using the same set of PPE simulations and a similar sensitivity analysis framework, we identify parameters that cause largest sensitivities over different regions and compare model responses in fast atmospheric processes to the parameters across different cloud regimes for several important cloud-related fidelity metrics. We find that cloud forcing has opposite response to some parameters over mid-latitude vs. tropical land. We also analyze how the parametric sensitivity varies as stratocumulus transitions to shallow convection and to deep convection over ocean. Low cloud forcing and shortwave cloud forcing in the subtropical eastern Pacific are most sensitive to the parameters controlling the width of the probability density function (PDF) of the subgrid vertical velocity ( w’ ) ( gamma ) and the damping of the w’ skewness ( c8 ) near the coast but become more sensitive to the parameter affecting the damping of the w’ variance ( c1 ) further offshore. Detailed interpretation of the spatial dependence of parametric sensitivity is provided. We also investigate how the parametric sensitivity evolves with prediction duration. This study improves our process-level understanding of cloud physics and parameterization and provides insights for developing more advanced regime-aware parameterization schemes in global climate model.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Anticipating how rain-on-snow events will change through the 21st century: lessons from the 1997 new year’s flood event

The California-Nevada 1997 New Year’s flood was an atmospheric river (AR)-driven rain-on-snow (RoS) event and remains the costliest in their history. The joint occurrence of saturated soils, rainfall, and snowmelt generated inundation throughout northern California-Nevada. Although AR RoS events are projected to occur more frequently with climate change, the warming sensitivity of their flood drivers across scales remains understudied. We leverage the regionally refined mesh capabilities of the Energy Exascale Earth System Model (RRM-E3SM) to recreate the 1997 New Year’s flood with horizontal grid spacings of 3.5 km across California, with forecast lead times of up to 4 days, and across six warming levels ranging from pre-industrial conditions to +3.5° C. We describe the sensitivity of the flood drivers to warming including AR duration and intensity, precipitation phase, intensity and efficiency, snowpack mass and energy changes, and runoff efficiency. Our findings indicate current levels of climate change negligibly influence the flood drivers. At warming levels ≥ 1.7° C, AR hazard potential increases, snowpack nonlinearly decreases, antecedent soil moisture decreases (except where the snowline retreats), and runoff decreases (except in the southern Sierra Nevada where antecedent snowpack persists). Storm total precipitation increases, but at rates below warming-induced increases in saturation-specific humidity. Warming intensifies short-duration, high-intensity rainfall, particularly where snowfall-to-rainfall transitions occur. This study highlights the nonlinear tradeoffs in 21st-century RoS flood hazards with warming and provides water management and infrastructure investment adaptation considerations.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2014

DIRAQ: scalable in situ data- and resource-aware indexing for optimized query performance

Scientific data analytics in high-performance computing environments has been evolving along with the advancement of computing capabilities. With the onset of exascale computing, the increasing gap between compute performance and I/O bandwidth has rendered the traditional post-simulation processing a tedious process. Despite the challenges due to increased data production, there exists an opportunity to benefit from "cheap" computing power to perform query-driven exploration and visualization during simulation time. To accelerate such analyses, applications traditionally augment, post-simulation, raw data with large indexes, which are then repeatedly utilized for data exploration. However, the generation of current state-of-the-art indexes involves a compute- and memory-intensive processing, thus rendering them inapplicable in an in situ context. In this paper we propose DIRAQ, a parallel in situ, in network data encoding and reorganization technique that enables the transformation of simulation output into a query-efficient form, with negligible runtime overhead to the simulation run. DIRAQ's effective core-local, precision-based encoding approach incorporates an embedded compressed index that is 3---6 $$\times $$ ? smaller than current state-of-the-art indexing schemes. Its data-aware index adjustmentation improves performance of group-level index layout creation by up to 35 % and reduces the size of the generated index by up to 27 %. Moreover, DIRAQ's in network index merging strategy enables the creation of aggregated indexes that speed up spatial-context query responses by up to $$10\times $$ 10 ? versus alternative techniques. DIRAQ's topology-, data-, and memory-aware aggregation strategy results in efficient I/O and yields overall end-to-end encoding and I/O time that is less than that required to write the raw data with MPI collective I/O.

Computer Science↗
DOE OSTI2023

Laboratory Study of Collisionless Magnetic Reconnection

A concise review is given on the past two decades’ results from laboratory experiments on collisionless magnetic reconnection in direct relation with space measurements, especially by the Magnetospheric Multiscale (MMS) mission. Highlights include spatial structures of electromagnetic fields in ion and electron diffusion regions as a function of upstream symmetry and guide field strength, energy conversion and partitioning from magnetic field to ions and electrons including particle acceleration, electrostatic and electromagnetic kinetic plasma waves with various wavelengths, and plasmoid-mediated multiscale reconnection. Combined with the progress in theoretical, numerical, and observational studies, the physics foundation of fast reconnection in collisionless plasmas has been largely established, at least within the parameter ranges and spatial scales that were studied. Immediate and long-term future opportunities based on multiscale experiments and space missions supported by exascale computation are discussed, including dissipation by kinetic plasma waves, particle heating and acceleration, and multiscale physics across fluid and kinetic scales.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2011

Designing Energy Efficient Communication Runtime Systems: A View from PGAS Models

As the march to the Exascale computing gains momentum, energy consumption of supercomputers has emerged to be the critical roadblock. While architectural innovations are imperative in achieving computing of this scale, it is largely dependent on the systems software to leverage the architectural innovations. Parallel applications in many computationally intensive domains have been designed to leverage these supercomputers, with legacy two-sided communication semantics using Message Passing Interface. At the same time, Partitioned Global Address Space Models are being designed which provide global address space abstractions and one-sided communication for exploiting data locality and communication optimizations. PGAS models rely on one-sided communication runtime systems for leveraging high-speed networks to achieve best possible performance. In this paper, we present a design for Power Aware One-Sided Communication Library - PASCoL. The proposed design detects communication slack, leverages Dynamic Voltage and Frequency Scaling (DVFS) and Interrupt driven execution to exploit the detected slack for energy efficiency. We implement our design and evaluate it using synthetic benchmarks for one-sided communication primitives, Put, Get and Accumulate and uniformly non-contiguous data transfers. Our performance evaluation indicates that we can achieve significant reduction in energy consumption without performance loss on multiple one-sided communication primitives. The achieved results are close to the theoretical peak available with the experimental test bed.

Vishnu, Abhinav↗
DOE OSTI2023

Evaluation of pre-training large language models on leadership-class supercomputers

Large language models (LLMs) have arisen rapidly to the center stage of artificial intelligence as the foundation models applicable to many downstream learning tasks. However, how to effectively build, train, and serve such models for many high-stake and first-principle-based scientific use cases are both of great interests and of great challenges. Moreover, pre-training LLMs with billions or even trillions of parameters can be prohibitively expensive not just for academic institutions, but also for well-funded industrial and government labs. Furthermore, the energy cost and the environmental impact of developing LLMs must be kept in mind. Here, in this work, we conduct a first-of-its-kind performance analysis to understand the time and energy cost of pre-training LLMs on the Department of Energy (DOE)’s leadership-class supercomputers. Employing state-of-the-art distributed training techniques, we evaluate the computational performance of various parallelization approaches at scale for a range of model sizes, and establish a projection model for the cost of full training. Our findings provide baseline results, best practices, and heuristics for pre-training such large models that should be valuable to HPC community at large. We also offer insights and optimization strategies for using the first exascale computing system, Frontier, to train models of the size of GPT-3 and beyond.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Scalable training of trustworthy and energy-efficient predictive graph foundation models for atomistic materials modeling: a case study with HydraGNN

We present our work on developing and training scalable, trustworthy, and energy-efficient predictive graph foundation models (GFMs) using HydraGNN, a multi-headed graph convolutional neural network architecture. HydraGNN expands the boundaries of graph neural network (GNN) computations in both training scale and data diversity. It abstracts over message passing algorithms, allowing both reproduction of and comparison across algorithmic innovations that define nearest-neighbor convolution in GNNs. This work discusses a series of optimizations that have allowed scaling up the GFMs training to tens of thousands of GPUs on datasets consisting of hundreds of millions of graphs. Our GFMs use multitask learning (MTL) to simultaneously learn graph-level and node-level properties of atomistic structures, such as energy and atomic forces. Using over 154 million atomistic structures for training, we illustrate the performance of our approach along with the lessons learned on two state-of-the-art US Department of Energy (US-DOE) supercomputers, namely the Perlmutter petascale system at the National Energy Research Scientific Computing Center and the Frontier exascale system at Oak Ridge Leadership Computing Facility. The HydraGNN architecture enables the GFM to achieve near-linear strong scaling performance using more than 2000 GPUs on Perlmutter and 16,000 GPUs on Frontier.

97 MATHEMATICS AND COMPUTING↗