Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “EXASCALE”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 991 records · Page 55

Simulation at Extreme-Scale: Co-Design Thinking and Practices

The current trend of computer architecture evolving towards exaflop/s is the fast increasing floating point performance (the so-called “free” flops) accompanied by much slowly improving the bandwidth of memory and network. Numerical simulation would undergo the challenge posed by the unbalanced increase in the compute power and the capability of data movement. In this paper, after reviewing the challenges of hardware and software in moving towards exascale computing, we present co-design thinking for selecting, optimizing, and developing a numerical algorithm and a simulation tool to meet the challenge of simulation at extreme scale. Examples are presented to demonstrate the new way of thinking and its effectiveness on the emerging architecture.

Computer Science↗

ESM data downscaling: a comparison of super-resolution deep learning models

Abstract Climate projections at fine spatial resolutions are required to conduct accurate risk assessment for critical infrastructure and design adaptation planning. Generating these projections using advanced Earth system models (ESM) requires significant computational resources. To address this issue, various statistical downscaling techniques have been introduced to generate fine-resolution data from coarse-resolution simulations. In this study, we evaluate and compare five deep learning-based downscaling techniques, namely, super-resolution convolutional neural networks, fast super-resolution convolutional neural network ESM, efficient sub-pixel convolutional neural network, enhanced deep residual network (EDRN), and super-resolution generative adversarial network (SRGAN). These techniques are applied to a dataset generated by the Energy Exascale Earth System Model (E3SM), focusing on key surface variables such as surface temperature, shortwave heat flux, and longwave heat flux. Models are trained and validated using paired fine-resolution (0.25 $$^{\circ }$$ ∘ ) and coarse-resolution (1 $$^{\circ }$$ ∘ ) monthly data obtained from a 9-year simulation. Next, blind testing is performed using monthly data obtained from two different years outside of the training and validation set. To evaluate the efficiency of each technique, different statistical metrics are used, including mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), and learned perceptual image patch similarity (LPIPS). The results show that EDRN outperforms other algorithms in terms of PSNR, SSIM, and MSE, but struggles to capture fine-scale features in the data. In contrast, SRGAN, a generative model that uses perceptual loss, excels in capturing fine details at boundaries and internal structures, resulting in lower LPIPS than other methods.

Pawar, Nikhil M. (ORCID:0000000211613289)↗

Evaluation and modification of ELM seasonal deciduous phenology against observations in a southern boreal peatland forest

Phenological transitions determine the timing of changes in land surface properties and the seasonality of exchanges of biosphere-atmosphere energy, water, and carbon. Accurate mechanistic modeling of phenological processes is therefore critical to understand and correctly predict terrestrial ecosystem feedbacks with changing atmospheric and climate conditions. However, the phenological components in the land model of the US Department of Energy's (DOE) Energy Exascale Earth System Model (ELM of E3SM) were previously unable to accurately capture the observed phenological responses to environmental conditions in a well-studied boreal peatland forest. In this research, we introduced new seasonal-deciduous phenology schemes into version 1.0 of ELM and evaluated their performance against the PhenoCam observations at the Spruce and Peatland Responses Under Changing Environments (SPRUCE) experiment in northern Minnesota from 2015 to 2018. We found that phenology simulated by the revised ELM (i.e., earlier spring onsets and stronger warming responses of spring onsets and autumn senescence) was closer to observations than simulations from the original algorithms for both the deciduous conifer (Larix laricina) and mixed shrub layers. Moreover, the revised ELM generally produced higher carbon and water fluxes (e.g., photosynthesis and evapotranspiration) during the growing season and stronger flux responses to warming than the default ELM. A parameter sensitivity analysis further indicated the significant contribution of phenology parameters to uncertainty in key carbon and water cycle variables, underscoring the importance of precise phenology parameterization. Furthermore, this phenological modeling effort demonstrates the potential to enhance the E3SM representation of land-climate interactions at broader spatiotemporal scales, especially under anticipated elevated CO 2 and warming conditions.

54 ENVIRONMENTAL SCIENCES↗

Evaluating ecosystem water use efficiency and recovery dynamics during flash droughts: insights from observations and model simulations

Flash droughts (FD), rapidly emerging in a warming future, disrupt ecosystems, agriculture, and water security. Ecosystem water use efficiency (WUE), the ratio of gross primary production (GPP) to actual evapotranspiration (AET), balances carbon assimilation and water loss. FD rapidly disrupts this balance, making WUE critical for assessing plant stress and recovery. Here, this study investigates the dynamics of landscape-scale WUE, and the components of GPP and AET under FD utilizing both observed data from the Missouri Ozark AmeriFlux site (US-MOz) and version 2 of the U.S. Department of Energy’s Earth, Energy, Exascale System Model (E3SM) Land Model (ELMv2). Observations and simulations reveal GPP as dominant for WUE during earlier FD events (2005, 2007, 2012), shifting to AET in recent events (2014, 2018). This agreement indicates that the ELM can capture the shifting dynamics of GPP and AET in regulating WUE under FD conditions. However, the ELM systematically underestimates both GPP and AET and does so in a manner that does not preserve their ratio. As a result, WUE is also underestimated, suggesting that GPP is more strongly underestimated than AET. Furthermore, the ELM also underestimates the speed of GPP recovery, producing an artificially prolonged GPP recovery time following FD events. Observed environmental drivers such as vapor pressure deficit (VPD), soil moisture (SM), and predawn leaf water potential (PLWP) effectively predict WUE, but ELM primarily highlights SM, underestimating VPD’s role. This study demonstrates that relying solely on soil moisture fails to capture the rapid hydraulic recovery observed in PLWP, underscoring the necessity of integrating plant hydraulics into land surface models to improve flash drought predictability.

Evapotranspiration↗

In situ multi-tier auto-ignition detection applied to dual-fuel combustion simulations

Here we use an anomaly detection methodology that is centered on analyzing fourth-order joint moments (co-kurtosis), particularly focusing on its application in auto-ignition of combustion problems with large numbers of species. Unsupervised anomaly detection is challenging to generalize across problem types and domains. A recent technique, centered on analyzing information in the fourth-order joint moment co-kurtosis, has shown promise, especially for high-dimensional scientific data. In this work we present developments to the co-kurtosis based anomaly detection method needed to make it effective and scalable for large-scale distributed scientific data, such as those generated by massively parallel simulations. An in situ co-kurtosis algorithm is employed as the anomaly detection method for identifying ignition kernels in simulations of turbulent combustion. Here, we extend an existing methodology which identifies regions of the domain where anomalies are present, and add another tier of anomaly detection where the individual samples contributing to the anomaly are identified. We apply this algorithm on-the-fly to a variety of turbulent reacting flow problems and compare it to the widely used (but significantly more expensive) chemical explosive mode analysis (CEMA). We demonstrate the ability of the method to detect and identify the onset of low and high temperature ignition which can be used for computational steering, as chemical and combustion anomalies occur intermittently at spatio-temporal locations unknown a priori. Finally, we apply our lightweight in situ algorithm to an exascale high-fidelity simulation with a total of 2.4 Trillion degrees of freedom, performed using an adaptive mesh refinement solver. Furthermore, through a scalability analysis, we show that the relative computational cost of this in-situ anomaly detection algorithm compared to an iteration of the reacting flow solver is negligible.

97 MATHEMATICS AND COMPUTING↗

Establishing mesh topology in multi-material cells: enabling technology for robust and accurate multi-material simulations

Real world problems are typically multi-material, combining materials such as gases, liquids and solids that have very different properties. The material interfaces may be fixed in time or can be a part of the solution, as in fluid-structure interactions or air-water dynamics, and therefore move and change shape. In such problems the computational mesh may be non-conformal to interfaces due to complexity of these interfaces, presence of small fractions of materials, or because the mesh does not move with the flow, as in the arbitrary Lagrangian–Eulerian (ALE) methods. In order to solve problems of interest on such meshes, interface reconstruction methods are usually used to recover an approximation of material regions within the cells. For a cell intersecting multiple material regions, these approximations of contained subregions can be considered as single-material subcells in a local mesh that we call a minimesh. In this paper, we discuss some of the requirements that discretization methods have on topological information in the resulting hierarchical meshes and present an approach that allows incorporating the buildup of sufficiently detailed topology into the nested dissections based PLIC-type reconstruction algorithms (e.g. Volume-of-Fluid, Moment-of-Fluid) in an efficient and robust manner. Specifically, we describe the X-MOF interface reconstruction algorithm in 2D, which extends the Moment-Of-Fluid (MOF) method to include the topology of minimeshes created inside of multi-material cells and parent-child relations between corresponding mesh entities on different hierarchy levels. X-MOF retains the property of being local to a cell and not requiring external communication, which makes it suitable for massively parallel applications. Here, we demonstrate some scaling results for the X-MOF implementation in Tangram, a modern interface reconstruction framework for exascale computing.

97 MATHEMATICS AND COMPUTING↗

GPU-accelerated DNS of compressible turbulent flows

Here, this paper explores strategies to transform an existing CPU-based high-performance computational fluid dynamics solver, HyPar, for compressible flow simulations on emerging exascale heterogeneous (CPU+GPU) computing platforms. The scientific motivation for developing a GPU-enhanced version of HyPar is to simulate canonical turbulent flows at the highest resolution possible on such platforms. We show that optimizing memory operations and thread blocks results in 200x speedup of computationally intensive kernels compared with a CPU core. Using multiple GPUs and CUDA-aware MPI communication, we demonstrate both strong and weak scaling of our GPU-based HyPar implementation on the NVIDIA Volta V100 GPUs. We simulate the decay of homogeneous isotropic turbulence in a triply periodic box on grids with up to 1024 3 points (5.3 billion degrees of freedom) and on up to 1,024 GPUs. We compare the wall times for CPU-only and CPU+GPU simulations. The results presented in the paper are obtained on the Summit and Lassen supercomputers at Oak Ridge and Lawrence Livermore National Laboratories, respectively.

97 MATHEMATICS AND COMPUTING↗

Highly-scalable GPU-accelerated compressible reacting flow solver for modeling high-speed flows

Emerging supercomputing systems utilize a combination of central processing units (CPUs) and graphics processing units (GPUs) in an effort to reach exascale capabilities while minimizing the energy footprint of operating such systems. Such heterogeneous machines introduce new challenges for fluids solvers because the hardware architecture and operation of a GPU are fundamentally different from conventional CPUs. In this work, a general approach for efficient implementation of finite-volume based reacting flow solvers on such heterogeneous systems is presented. Three main challenges, namely, data access pattern, thread divergence, and thread safety, are addressed. Since compressible reacting flows require special methods to deal with chemical reactions, hyperbolic and nonlinear convection terms, and the presence of turbulence, specific algorithms that ensure GPU-based efficiency are developed. The approach is demonstrated on the widely available OpenFOAM open source software by modifying core algorithms for GPU accessibility. The scalability of the resulting solver, is demonstrated using practical test cases, including flow through a scramjet engine and the dynamics of a rotating detonation engine. Here, the solver provides near-ideal scaleup on a large number of GPUs (>3000), and extremely efficient use of the GPUs, with throughput nearly a constant even when processing a large number of control volumes.

42 ENGINEERING↗

MPI + OpenACC: Accelerating radiation transport mini-application, minisweep, on heterogeneous systems

According to the article, architectures are rapidly evolving, and exascale machines are expected to offer billion-way concurrency. We need to rethink algorithms, languages and programming models among other components in order to migrate large scale applications and explore parallelism on these machines. Although directive-based programming models allow programmers to worry less about programming and more about science, expressing complex parallel patterns in these models can be a daunting task especially when the goal is to match the performance that the hardware platforms can offer. One such pattern is wavefront. This paper extensively studies a wavefront-based miniapplication for Denovo, a production code for nuclear reactor modeling. We parallelize the Koch–Baker–Alcouffe (KBA) parallel-wavefront sweep algorithm in the main kernel of Minisweep (the miniapplication) using CUDA 9.0, OpenMP 4.0 (SIMD) and OpenACC 2.6. Our OpenACC implementation running on NVIDIA’s next-generation Volta GPU boasts an 85.06x speedup over serial code, which is larger than CUDA’s 83.72x speedup over the same serial implementation. We also explore the scalability of our solution using MPI to decompose our simulation domain, allowing us to run on many nodes and accelerators present in state-of-the-art HPC systems. Our parallelization effort across platforms also motivated us to define an abstract parallelism model that is architecture independent, with a goal of creating software abstractions that can be used by applications employing the wavefront sweep motif.

Computer Science↗

A generalized massively parallel ultra-high order FFT-based Maxwell solver

Dispersion-free ultra-high order FFT-based Maxwell solvers have recently proven to be paramount to a large range of applications, including the high-fidelity modeling of high-intensity laser–matter interactions with Particle-In-Cell (PIC) codes. To enable a massively parallel scaling of these solvers, a novel parallelization technique was recently proposed, which consists in splitting the simulation domain into several processor sub-domains, with guard regions appended at each sub-domain boundary. Maxwell's equations are advanced independently on each sub-domain using local shared-memory FFTs (instead of a single distributed global FFT). This implies small truncation errors at sub-domain boundaries, the amplitude of which depends on guard regions sizes and order of the Maxwell solver. For moderate guard region sizes, this ’local’ technique proved to be highly scalable on up to a million cores and notably enabled the 3D modeling of so-called plasma mirrors, for which 8 guard cells only were enough to prevent truncation error growth. Yet, for other applications, the required number of guard cells might be much higher, which would severely limit the parallel efficiency of this technique due to the large volume of guard cells to be exchanged between sub-domains. In this context, we propose a novel parallelization technique that ensures very good scaling of FFT-based solvers with an arbitrarily high number of guard cells. Our ’hybrid’ technique consists in performing distributed FFTs on local groups of processors with guard regions now appended to boundaries of each group of processors. It uses a dual domain decomposition method for the Maxwell solver and other parts of the PIC cycle to keep the simulation load-balanced. This ’hybrid’ technique was implemented in the open source exascale library PICSAR. Benchmarks show that for a large number of guard cells (>16), the ’hybrid’ technique offers up to ×3 speed-up and ×8 memory savings compared to the ’local’ one.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Dynamic load balancing with enhanced shared-memory parallelism for particle-in-cell codes

Furthering our understanding of many of today’s interesting problems in plasma physics – including plasma based acceleration and magnetic reconnection with pair production due to quantum electrodynamic effects – requires large-scale kinetic simulations using particle-in-cell (PIC) codes. However, these simulations are extremely demanding, requiring that contemporary PIC codes be designed to efficiently use a new fleet of exascale computing architectures. To this end, the key issue of parallel load balance across computational nodes must be addressed. We discuss the implementation of dynamic load balancing by dividing the simulation space into many small, self-contained regions or ‘‘tiles,’’ along with shared-memory (e.g., OpenMP) parallelism both over many tiles and within single tiles. The load balancing algorithm can be used with three different topologies, including two space-filling curves. Here, we tested this implementation in the code Osiris and show low overhead and improved scalability with OpenMP thread number on simulations with both uniform load and severe load imbalance. Compared to other load-balancing techniques, our algorithm gives order-of-magnitude improvement in parallel scalability for simulations with severe load imbalance issues.

97 MATHEMATICS AND COMPUTING↗

An OpenMP GPU-offload implementation of a non-equilibrium solidification cellular automata model for additive manufacturing

Here, in this paper, performance strategies on GPU-based HPC platforms of a cellular automata (CA) simulation code for non-equilibrium solidification, including nucleation, grain growth, solute partitioning and transport for the metal additive manufacturing (AM) process are investigated using OpenMP 4.5. To accurately report the speed-up for multicore CPUs and GPUs, a rigorous performance analysis employed optimizations appropriate for both CPU-only code (baseline) and GPU offload codes for an isothermal test problem. The performance results on Summit at the Oak Ridge Leadership Computing Facility indicate that using a precomputed list of interface cells significantly decreased the wall-clock time on GPUs. The speedup due to GPU acceleration was evaluated for a full Summit node and measured to be 1.8X when comparing a 6 MPI tasks run with 6 GPUs versus 36 MPI tasks on the CPU only. That speed-up was found to be 7.9X when comparing 6 MPI tasks with 6 GPUs versus the 6 MPI tasks running on the CPU only. Performance measurements showed that system total time is almost constant for runs with more than 96 MPI tasks (or GPUs), indicating that the GPU-accelerated code showed an excellent weak scaling performance. Finally, a rapid directional solidification problem was considered to demonstrate the CA code capability on Summit. It was found that a mesh size of at least 0.05 μm is recommended for the AM-like simulations in order to obtain accurate elongated grain microstructure and elongated subgrain features, which are in qualitative good agreement with experimental data. The results presented in this study indicate that the performance strategies on GPU-based HPC platforms for the CA code are appropriate for novel HPC exascale platforms.

36 MATERIALS SCIENCE↗

SIMULATeQCD: A simple multi-GPU lattice code for QCD calculations

The rise of exascale supercomputers has fueled competition among GPU vendors, driving lattice QCD developers to write code that supports multiple APIs. Moreover, new developments in algorithms and physics research require frequent updates to existing software. These challenges have to be balanced against constantly changing personnel. At the same time, there is a wide range of applications for HISQ fermions in QCD studies. This situation encourages the development of software featuring a HISQ action that is flexible, high-performing, open source, easy to use, and easy to adapt. In this technical paper, we explain the design strategy, provide implementation details, list available algorithms and modules, and show key performance indicators for SIMULATeQCD, a simple multi-GPU lattice code for large-scale QCD calculations, mainly developed and used by the HotQCD collaboration. The code is publicly available on GitHub.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

A parallel and performance portable implementation of a full-field crystal plasticity model

We have developed a parallel implementation of an Elasto-Viscoplastic Fast Fourier Transform-based (EVPFFT) micromechanical solver to enable computationally efficient crystal plasticity modeling for polycrystalline materials. Our primary focus lies in achieving performance portability, allowing a single EVPFFT implementation to run optimally on various homogeneous architectures, including multi-core Central Processing Units (CPUs), as well as on heterogeneous computer architectures comprising multi-core CPUs and Graphics Processing Units (GPUs) from different vendors. To accomplish this goal, we have leveraged MATAR, a C++ software library that simplifies the creation and utilization of multidimensional dense or sparse matrix and array data structures. These data structures are designed to be portable across diverse architectures through the use of Kokkos, a performance-portable library. Additionally, we have employed the Message Passing Interface (MPI) to efficiently distribute the computational workload among processors. The heFFTe (Highly Efficient FFT for Exascale) library is used to facilitate the performance portability of the fast Fourier transforms (FFTs) computation. The computational performance of EVPFFT is evaluated and presented in terms of parallel scalability and simulation runtime on different high-performance computing (HPC) architectures. As a result, the utility of the developed framework to efficiently simulate the micro-mechanical fields in polycrystalline microstructures in engineering applications is discussed.

36 MATERIALS SCIENCE↗

Considering coasts: Adapting terrestrial models to characterize coastal wetland ecosystems

The Energy Exascale Earth System Model (E3SM) simulates fully coupled processes and interactions among water, energy, carbon and nutrient cycles. E3SM connects vegetation and soil dynamics through nutrient uptake, plant production, litterfall and decomposition as a function of abiotic parameters (e.g. temperature and moisture). However, E3SM is designed to characterize terrestrial ecosystems and connects land and open ocean systems using a single streamflow transport term, ignoring the complex dynamics of energy, water, carbon, and nutrients in coastal systems. The goals of our project were to: (1) Parameterize a point version of E3SM to capture coastal wetland habitats and (2) Determine marsh community responses to increased temperature and elevated CO 2 . We adapted a version of the E3SM land model, previously configured to represent forested bog hydrology to a coastal ecosystem using datasets from field experiments conducted at the Smithsonian Environmental Research Center's Global Change Research Wetland (GCReW). Tidal forcing in a marsh environment was simulated using a two-column system in which the columns are connected by lateral hydrologic flows. One column simulates interactions between vegetation and soil while a second column simulates variation in water level (both tidal and sea level rise). The updated model captures many aspects of the field experiments, showing that plant community responses to environmental change are non-linear, non-additive and different between plant types. Elevated CO 2 treatments increased C 3 plant biomass more than C 4 (33% vs 17%). Temperature exacerbated CO 2 responses in C 3 plants (0 °C: 26%, 5.1 °C: 56%). We were more successful at characterizing C 3 than C 4 responses and simulating above rather than belowground biomass production. Furthermore, the next steps will include updates to key physiological parameters such as root:shoot carbon allocation and the addition of mechanistic feedbacks between vegetation and biogeochemical processes.

54 ENVIRONMENTAL SCIENCES↗

Computational homogenization at extreme scales

Multi-scale simulations at extreme scales in terms of both physical length scales and computational resources are presented. In this letter, we introduce a hierarchically parallel computational homogenization solver that employs hundreds of thousands of computing cores and resolves in material length scales (from to ). Simulations of this kind are important in understanding the multi-scale essence of many natural and synthetically made materials. Thus, we present a simulation consisting of 53.8 Billion finite elements with 28.1 Billion nonlinear equations that is solved on 393,216 computing cores (786,432 threads). The excellent parallel performance of the computational homogenization solver is demonstrated by a strong scaling test from 4,096 to 262,144 cores. A fully coupled multi-scale damage simulation shows a complex crack profile at the micro-scale and the macroscopic crack tunneling phenomenon. Such large and predictive simulations are an important step towards Virtual Materials Testing and can aid in development of new material formulations with extreme properties. Furthermore, the high computational efficiency of our computational homogenization solver holds great promise for utilizing the next generation of exascale parallel computing platforms that are expected to accelerate computations through orders of magnitude increase in parallelism rather than speed of each processor.

Engineering↗

A Comprehensive Chemistry Evaluation and Diagnostics Package for E3SM – ChemDyg Version 1.1.0

The Chemistry Evaluation and Diagnostics Package (ChemDyg) is an open-source tool designed for the Energy Exascale Earth System Model (E3SM) developed by the U.S. Department of Energy. ChemDyg facilitates routine evaluation, tailored development, and in-depth analysis of atmospheric chemistry through its modular architecture, allowing users to compare model outputs with observational data. Version 1.1.0 introduces a robust set of diagnostic capabilities, including climatology, time evolution of key tracers, diurnal and annual cycle analyses, and extensive budget diagnostics. These features help identify model discrepancies and enhance the representation of atmospheric chemistry in E3SM. Each self-contained diagnostic set includes dedicated scripts and documentation for ease of use. The interactive HTML output improves data accessibility, accelerating chemistry model development. Additionally, ChemDyg's flexible framework allows for customization, enabling users to create unique diagnostic sets for specific scientific contributions.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Numerical analysis of particle dispersion and deposition in coal combustion using large-eddy simulation

High-fidelity modeling provides a useful approach to investigate the particle dispersion and deposition mechanism in pulverized coal combustion. To be able to analyze these detailed mechanisms, this work couples detailed multiphysics models for poly-dispersed turbulent, particle-laden flow, particle and gas phase reaction chemistry, convective, conductive and radiative heat transfer, ash formation and deposition with a Large-Eddy Simulation (LES) approach to numerically simulate coal combustion in a downfired self-sustained oxy-fuel combustor (OFC). It is necessary to explicitly capture all but the highest frequency dynamics of the turbulence and its coupling with each of the other physical phenomena using LES. Effects of subgrid-scale unresolved turbulence on the particle motions are also analyzed by a novel Stokes number analysis. Due to the inherent relation ship between coal combustion and ash deposition, this study integrates the improved ash deposition model into the numerical simulation. Overall simulation results are compared with experimentally measured data from the OFC. The simulation and measured data for the averaged gas temperature and deposition rates agree with 5% and 28%. This study shows that high-fidelity LES coupled with other detailed multiphysics models running on a exascale computing facility can provide a good representation of complex coal combustion and deposition in a laboratory-scale furnace.

multiphysics model↗