Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “out of memory”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,045 records · Page 58
All sourcesNASA NTRSDOE OSTI
DOE OSTI2020

High-order matrix-free incompressible flow solvers with GPU acceleration and low-order refined preconditioners

In this work, we present a matrix-free flow solver for high-order finite element discretizations of the incompressible Navier-Stokes and Stokes equations with GPU acceleration. For high polynomial degrees, assembling the matrix for the linear systems resulting from the finite element discretization can be prohibitively expensive, both in terms of computational complexity and memory. For this reason, it is necessary to develop matrix-free operators and preconditioners, which can be used to efficiently solve these linear systems without access to the matrix entries themselves. The matrix-free operator evaluations utilize GPU-accelerated sum-factorization techniques to minimize memory movement and maximize throughput. The preconditioners developed in this work are based on a low-order refined methodology with parallel subspace corrections, as described for diffusion problems in [1]. The saddle-point Stokes system is solved using block-preconditioning techniques, which are robust in mesh size, polynomial degree, time step, and viscosity. For the incompressible Navier-Stokes equations, we make use of projection (fractional step) methods, which require Helmholtz and Poisson solves at each time step. The performance of our flow solvers is assessed on several benchmark problems in two and three spatial dimensions.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Efficient discontinuous Galerkin implementations and preconditioners for implicit unsteady compressible flow simulations

This work presents and compares efficient implementations of high-order discontinuous Galerkin methods: a modal matrix-free discontinuous Galerkin (DG) method, a hybridizable discontinuous Galerkin (HDG) method, and a primal formulation of HDG, applied to the implicit solution of unsteady compressible flows. The matrix-free implementation allows for a reduction of the memory footprint of the solver when dealing with implicit time-accurate discretizations. HDG reduces the number of globally-coupled degrees of freedom relative to DG, at high order, by statically condensing element-interior degrees of freedom from the system in favor of face unknowns. The primal formulation further reduces the element-interior degrees of freedom by eliminating the gradient as a separate unknown. This paper introduces a p-multigrid preconditioner implementation for these discretizations and presents results for various flow problems. Benefits of the p-multigrid strategy relative to simpler, less expensive, preconditioners are observed for stiff systems, such as those arising from low-Mach number flows at high-order approximation. The p-multigrid preconditioner also shows excellent scalability for parallel computations. Finally, additional savings in both speed and memory occur with a matrix-free/reduced version of the preconditioner.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

GPU-enabled extreme-scale turbulence simulations: Fourier pseudo-spectral algorithms at the exascale using OpenMP offloading

Fourier pseudo-spectral methods for nonlinear partial differential equations are of wide interest in many areas of advanced computational science, including direct numerical simulation of three-dimensional (3-D) turbulence governed by the Navier-Stokes equations in fluid dynamics. This paper presents a new capability for simulating turbulence at a new record resolution up to 35 trillion grid points, on the world's first exascale computer, Frontier, comprising AMD MI250x GPUs with HPE's Slingshot interconnect and operated by the US Department of Energy's Oak Ridge Leadership Computing Facility (OLCF). Key programming strategies designed to take maximum advantage of the machine architecture involve performing almost all computations on the GPU which has the same memory capacity as the CPU, performing all-to-all communication among sets of parallel processes directly on the GPU, and targeting GPUs efficiently using OpenMP offloading for intensive number-crunching including 1-D Fast Fourier Transforms (FFT) performed using AMD ROCm library calls. With 99% of computing power on Frontier being on the GPU, leaving the CPU idle leads to a net performance gain via avoiding the overhead of data movement between host and device except when needed for some I/O purposes. Memory footprint including the size of communication buffers for MPI_ALLTOALL is managed carefully to maximize the largest problem size possible for a given node count. Detailed performance data including separate contributions from different categories of operations to the elapsed wall time per step are reported for five grid resolutions, from 2048 3 on a single node to 32768 3 on 4096 or 8192 nodes out of 9408 on the system. Both 1D and 2D domain decompositions which divide a 3D periodic domain into slabs and pencils respectively are implemented. The present code suite (labeled by the acronym GESTS, GPUs for Extreme Scale Turbulence Simulations) achieves a figure of merit (in grid points per second) exceeding goals set in the Center for Accelerated Application Readiness (CAAR) program for Frontier. The performance attained is highly favorable in both weak scaling and strong scaling, with notable departures only for 2048 3 where communication is entirely intra-node, and for 32768 3 , where a challenge due to small message sizes does arise. Communication performance is addressed further using a lightweight test code that performs all-to-all communication in a manner matching the full turbulence simulation code. Performance at large problem sizes is affected by both small message size due to high node counts as well as dragonfly network topology features on the machine, but is consistent with official expectations of sustained performance on Frontier. Overall, although not perfect, the scalability achieved at the extreme problem size of 32768 3 (and up to 8192 nodes — which corresponds to hardware rated at just under 1 exaflop/sec of theoretical peak computational performance) is arguably better than the scalability observed using prior state-of-the-art algorithms on Frontier's predecessor machine (Summit) at OLCF. New science results for the study of intermittency in turbulence enabled by this code and its extensions are to be reported separately in the near future.

3D fast Fourier transform↗
DOE OSTI2020

The effect of lighting environment on task performance in buildings – A review

The effects of indoor environmental conditions on human health, satisfaction, and performance have been the focal point of research for decades. This paper reviews and summarizes the impact of lighting environment on task performance, specifically for the built environment audience. Existing studies included a variety of performance tests on cognitive performance and perception, visual acuity and reaction, memory, reasoning, and labor productivity. Illuminance, luminance ratio and correlated color temperature were found to affect performance in different ways, reflecting the impact of experimental techniques, conditions, performance evaluation methods used and data analysis methods. These were reviewed and categorized, with discussion on limitations related to sample size, modeling approach, carryover effects and other factors affecting individual differences in performance, with recommendations for future improvement. Although no universal conclusions can be made, in general, task performance seems to improve with higher illuminances, contrast ratios in the range of 7–11:1 (while always making sure that glare will not occur in the space) and higher correlated color temperature, while spectral tuning in the red or blue wavelengths has also shown positive effects. To obtain more generic evidence, future studies should be more consistent in terms of experimental procedures and overall light conditions, and also consider the effects of vertical illuminance, daylight provision/control, and outside views on task performance. Finally, studying performance with multi-factorial designs in a human-centered optimized manner (such as deploying variable lighting scenarios optimized for various tasks) can lead to deeper understanding of lighting effects on task performance, and ultimately to improved lighting design and operation in buildings overall.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2024

A GPU accelerated mixed-precision Smoothed Particle Hydrodynamics framework with cell-based relative coordinates

Smoothed Particle Hydrodynamics (SPH) is essential for modeling complex large-deformation problems across various applications, requiring significant computational power. A major portion of SPH computation time is dedicated to the Nearest Neighboring Particle Search (NNPS) process. While advanced NNPS algorithms have been developed to enhance SPH efficiency, the potential efficiency gains from modern computation hardware remain underexplored. Here, this study investigates the impact of GPU parallel architecture, low-precision computing on GPUs, and GPU memory management on NNPS efficiency. Our approach employs a GPU-accelerated mixed-precision SPH framework, utilizing low precision float-point 16 (FP16) for NNPS while maintaining high precision for other components. To ensure FP16 accuracy in NNPS, we introduce a Relative Coordinated-based Link List (RCLL) algorithm, storing FP16 relative coordinates of particles within background cells. Our testing results show three significant speedup rounds for CPU-based NNPS algorithms. The first comes from parallel GPU computations, with up to a 1000x efficiency gain. The second is achieved through low-precision GPU computing, where the proposed FP16-based RCLL algorithm offers a 1.5x efficiency improvement over the FP64-based approach on GPUs. By optimizing GPU memory bandwidth utilization, the efficiency of the FP16 RCLL algorithm can be further boosted by 2.7x, as demonstrated in an example with 1 million particles. Our code is released at https://github.com/pnnl/lpNNPS4SPH.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Air compressor load forecasting using artificial neural network

Air compressor systems are responsible for approximately 10% of the electricity consumed in United States and European Union industry. As many researches have proven the effectiveness of using Artificial Neural Network inair compressor performance prediction, there is still a need to forecast the air compressor electrical load profile. The objective of this study is to predict compressed air systems’ electrical load profile, which is valuable to industry practitioners as well as software providers in developing better practice and tools for load managementand look-ahead scheduling programs. Two artificial neural networks, Two-Layer Feed-Forward Neural Networkand Long Short-Term Memory were used to predict an air compressors electrical load. Compressors with three different control mechanisms are evaluated with a total number of 11,874 observations. Here, the forecasts were validated using out-of-sample datasets with 5-fold cross-validation. Models produced average coefficient ofdetermination values from 0.24 to 0.94, average root-mean-square errors from 0.05 kW - 5.83 kW, and meanabsolute scaled errors from 0.20 to 1.33. The results indicate that both artificial neural networks yield goodresults for compressors using variable speed drive (average R 2 = 0.8 and no naïve forecasting), only the longshort-term memory model gives acceptable results for compressors using on/off control (average R 2 = 0.82 and no naïve forecasting), and no satisfactory results are obtained for load/unload type air compressors (models constituting active forecasting).

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗
DOE OSTI2023

Quantitative proton radiography and shadowgraphy for arbitrary intensities

Charged-particle radiography and shadowgraphy data can be directly inverted to obtain a line-integrated transverse Lorentz force or a line-integrated transverse refractive index gradient if intensity modulations due to scattering and absorption are negligible, and angular deflections are small. We develop a new direct-inversion algorithm based on plasma physics and compare it to a new Monge–Ampère code and an existing power diagram code. The measured or source intensity is represented by electrons subject to drag, and the other intensity by fixed ions. The decrease in kinetic plus electrostatic energy determines convergence. The displacement of the electrons from their initial to their equilibrium positions determines the line-integrated force or refractive index gradient. We have implemented two approaches: PIC (particle in cell) and Lagrangian fluid, in 1-D and 2-D. The PIC code works for arbitrary intensities, can work efficiently in parallel, and can make use of existing codes. The Lagrangian code requires less memory and is faster than the PIC code without massively parallel processing, but fails in 2-D for large intensity modulations. The Monge–Ampère code is by far the fastest in 2-D, without massively parallel processing, but fails for intensities with large voids, high contrast ratios and large deflections across the boundaries, and could not obtain the degree of convergence possible with the PIC code. As a result, the power diagram code was by far the slowest and most memory intensive, and failed for large peaks in the measured intensity.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2020

Development of a CPU/GPU portable software library for Lagrangian–Eulerian simulations of liquid sprays

The Lagrangian–Eulerian method is widely used for simulations of fuel sprays in turbulent combustion because of the advantage of treating the spray droplets as discrete points. One challenge of the Lagrangian–Eulerian method is the intense computational requirement when tracking the large number of Lagrangian particles needed for high fidelity. We have developed a performance-portable library, Grit, to track the Lagrangian particles in parallel on central processing unit (CPU) and graphics processing unit (GPU) accelerated high performance computing (HPC) architectures. Grit is a C++ library which employs Message Passing Interface (MPI) for distributed memory parallelism and Kokkos programming model for on-node shared memory parallelism with performance portability across different architectures of GPUs and multi-core/manycore CPUs. The parallel algorithms, key parallel kernels, and their performances on the pre-exascale supercomputer, Summit, are presented. Grit is coupled with a direct numerical simulation (DNS) solver, S3D, for multiphase simulations. A conservative formulation has been developed and implemented in Grit for phase coupling with thermodynamic consistency. The formulation separates the conservation of mass, momentum and energy from the physical models to prevent accidental violation of conservation laws due to inconsistent models. The formulation also enforces consistent definitions of enthalpies of the fuel for both phases and the latent heat of evaporation. Finally, simulations of turbulent particle-laden flow and the evaporation of dilute turbulent spray jet are performed to verify the software implementation, and to demonstrate the scalability of Grit for large-scale multiphase simulations.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Nitrogen: A promising doping strategy for high-performance ovonic threshold switching selectors

The Ovonic Threshold Switching (OTS) selector serves as an essential component in the development of three-dimensional high-density memory integration technology. Nevertheless, the state-of-the-art high-performance OTS materials usually contain toxic elements such as arsenic (As), posing significant risks to both environmental and human health. Nitrogen (N), which belongs to the same group as arsenic (As), has emerged as a highly promising alternative for As doping. However, the underlying mechanisms that govern N-based OTS materials have not yet been extensively investigated. In this study, we delve into the effects of N doping on the structural, bonding, and electronic properties of amorphous GeSe (a-GeNSe) by ab initio molecular dynamics simulations to bridge the knowledge gap. Our findings indicate that upon N doping in a-GeSe, the formation of robust Ge-N bonds, along with N-centered tetrahedral and triangular structures, resulting in the sluggish atomic movement that enhances the thermal stability and endurance of a-GeNSe. The OTS characteristics are significantly influenced by the material’s electronic band structure, and thus the relatively slow performance drift can be attributed to the stabilization of mid-gap states, a result of N doping which effectively slows down the aging process of chalcogenide glass. Moreover, the increased mobility gap in a-GeNSe raises the threshold voltage (V th ), making it more compatible with commercially available phase-change memory materials. Furthermore, our findings reveal the extensive impact of the N element on a typical OTS material and offer valuable perspectives for alternative doping strategies that could potentially supplant As practices.

36 MATERIALS SCIENCE↗
DOE OSTI2020

Pass-efficient methods for compression of high-dimensional turbulent flow data

The future of high-performance computing, specifically on future Exascale computers, will presumably see memory capacity and bandwidth fail to keep pace with data generated, for instance, from massively parallel partial differential equation (PDE) systems. Current strategies proposed to address this bottleneck entail the omission of large fractions of data, as well as the incorporation of in situ compression algorithms to avoid overuse of memory. To ensure that post-processing operations are successful, this must be done in a way that a sufficiently accurate representation of the solution is stored. Moreover, in situations where the input/output system becomes a bottleneck in analysis, visualization, etc., or the execution of the PDE solver is expensive, the number of passes made over the data must be minimized. In the interest of addressing this problem, this work focuses on the utility of pass-efficient, parallelizable, low-rank, matrix decomposition methods in compressing high-dimensional simulation data from turbulent flows. Additionally, a particular emphasis is placed on using coarse representation of the data – compatible with the PDE discretization grid – to accelerate the construction of the low-rank factorization. This includes the presentation of a novel single-pass matrix decomposition algorithm for computing the so-called interpolative decomposition. The methods are described extensively and numerical experiments on two turbulent channel flow data are performed. In the first (unladen) channel flow case, compression factors exceeding 400 are achieved while maintaining accuracy with respect to first- and second-order flow statistics. In the particle-laden case, compression factors of 100 are achieved and the compressed data is used to recover particle velocities. These results show that these compression methods can enable efficient computation of various quantities of interest in both the carrier and disperse phases.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2021

Higher-order particle representation for particle-in-cell simulations

In this paper we present an alternative approach to the representation of simulation particles for unstructured electrostatic and electromagnetic PIC simulations. In our modified PIC algorithm we represent particles as having a smooth shape function limited by some specified finite radius, r 0 . A unique feature of our approach is the representation of this shape by surrounding simulation particles with a set of virtual particles with delta shape, with fixed offsets and weights derived from Gaussian quadrature rules and the value of r 0 . As the virtual particles are purely computational, they provide the additional benefit of increasing the arithmetic intensity of traditionally memory bound particle kernels. The modified algorithm is implemented within Sandia National Laboratories' unstructured EMPIRE-PIC code, for electrostatic and electromagnetic simulations, using periodic boundary conditions. We show results for a representative set of benchmark problems, including electron orbit, a transverse electromagnetic wave propagating through a plasma, numerical heating, and a plasma slab expansion. In this work, good error reduction across all of the chosen problems is achieved as the particles are made progressively smoother, with the optimal particle radius appearing to be problem-dependent.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗
DOE OSTI2022

Spectral quadrature for the first principles study of crystal defects: Application to magnesium

In this work, we present an accurate and efficient finite-difference formulation and parallel implementation of Kohn-Sham Density (Operator) Functional Theory (DFT) for non periodic systems embedded in a bulk environment. Specifically, employing non-local pseudopotentials, local reformulation of electrostatics, and truncation of the spatial Kohn-Sham Hamiltonian, and the Linear Scaling Spectral Quadrature method to solve for the pointwise electronic fields in real-space and the non-local component of the atomic force, we develop a parallel finite difference framework suitable for distributed memory computing architectures to simulate non-periodic systems embedded in a bulk environment. Choosing examples from magnesium-aluminum alloys, we first demonstrate the convergence of energies and forces with respect to spectral quadrature polynomial order, and the width of the spatially truncated Hamiltonian. Next, we demonstrate the parallel scaling of our framework, and show that the computation time and memory scale linearly with respect to the number of atoms. Next, we use the developed framework to simulate isolated point defects and their interactions in magnesium-aluminum alloys. Our findings conclude that the binding energies of divacancies, Al solute-vacancy and two Al solute atoms are anisotropic and are dependent on cell size. Furthermore, the binding is favorable in all three cases.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

A low-rank power iteration scheme for neutron transport criticality problems

Computing effective eigenvalues for neutron transport often requires a fine numerical resolution. Here, the main challenge of such computations is the high memory effort of classical solvers, which limits the accuracy of chosen discretizations. In this work, we derive a method for the computation of effective eigenvalues when the underlying solution has a low-rank structure. This is accomplished by utilizing dynamical low-rank approximation (DLRA), which is an efficient strategy to derive time evolution equations for low-rank solution representations. The main idea is to interpret the iterates of the classical inverse power iteration as pseudo-time steps and apply the DLRA concepts in this framework. In our numerical experiment, we demonstrate that our method significantly reduces memory requirements while achieving the desired accuracy. Analytic investigations show that the proposed iteration scheme inherits the convergence speed of the inverse power iteration, at least for a simplified setting.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Moment-based adaptive time integration for thermal radiation transport

Here, in this paper we develop a framework for moment-based adaptive time integration of deterministic multifrequency thermal radiation transpot (TRT). We generalize our recent semi-implicit-explicit (IMEX) integration framework for gray TRT to multifrequency TRT, and also introduce a semi-implicit variation that facilitates higher-order integration of TRT, where each stage is implicit in all components except opacities. To appeal to the broad literature on adaptivity with Runge–Kutta methods, we derive new embedded methods for four asymptotic preserving IMEX Runge–Kutta schemes we have found to be robust in our previous work on TRT and radiation hydrodynamics. We then use a moment-based high-order-low-order representation of the transport equations. Due to the high dimensionality, memory is always a concern in simulating TRT. We form error estimates and adaptivity in time purely based on temperature and radiation energy, for a trivial overhead in computational cost and memory usage compared with the base second order integrators. We then test the adaptivity in time on the tophat and Larsen problem, demonstrating the ability of the adaptive algorithm to naturally vary the timestep across 4–5 orders of magnitude, ranging from the dynamical timescales of the streaming regime to the thick diffusion limit.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Kernel fusion in atomistic spin dynamics simulations on Nvidia GPUs using tensor core

In atomistic spin dynamics simulations, the time cost of constructing the space- and time-displaced pair correlation function in real space increases quadratically as the number of spins N, leading to significant computational effort. The GEMM subroutine can be adopted to accelerate the calculation of the dynamical spin-spin correlation function, but the computational cost of simulating large spin systems (>40000 spins) on CPUs remains expensive. In this work, we perform the simulation on the graphics processing unit (GPU), a hardware solution widely used as an accelerator for scientific computing and deep learning. Here we show that GPUs can accelerate the simulation up to 25-fold compared to multi-core CPUs when using the GEMM subroutine on both. To hide memory latency, we fuse the element-wise operation into the GEMM kernel using CUTLASS that can improve the performance by 26% ~ 33% compared to implementation based on cuBLAS. Furthermore, we perform the on-the-fly calculation in the epilogue of the GEMM subroutine to avoid saving intermediate results on global memory, which makes the large-scale atomistic spin dynamics simulation feasible and affordable.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2021

A PIPS + SrI 2 (Eu) detector for atmospheric radioxenon monitoring

The PIPS–SrI 2 (Eu) is a prototype atmospheric radioxenon detection system designed at Oregon State University in support of international efforts towards monitoring clandestine nuclear weapon testing activities. This detector aims to address some shortcomings found in currently deployed beta–gamma atmospheric radioxenon detection systems, such as lackluster energy resolution and memory effect, by employing modern detection materials and readout. The system uses a PIPSBox, a silicon-based gas cell, for electron detection, and a pair of ultrabright, D-shaped SrI 2 (Eu) scintillators coupled to silicon photomultipliers for photon detection. A custom eight-channel digital pulse processor equipped with a field programmable gate-array (FPGA) identifies electron–photon coincidences between the volumes in near real-time. Gas samples of the four radioxenon isotopes of interest were independently measured with the PIPS–SrI 2 (Eu) detection system to determine energy resolution and efficiency. Application of FPGA-based coincidence discrimination in near real-time reduced the ambient background count rate by 95.85 ± 0.04%. Using parameters from the Xenon International gas processing unit and assuming a blank sample and zero memory effect the minimum detectable concentrations (MDCs) for the isotopes were calculated to be 0.12 ± 0.03, 0.27 ± 0.05, 0.15 ± 0.02, and 1.00 ± 0.08 mBq/m 3 air for 131m Xe, 133 Xe, 133m Xe, and 135 Xe, respectively. These MDC estimates compare well with other radioxenon detection systems employed in the International Monitoring System (IMS) and indicate that the PIPS–SrI 2 (Eu) is in compliance with the Comprehensive Nuclear Test-Ban-Treaty Organization (CTBTO) sensitivity requirement of ≤ 1 mBq/m 3 for 133 Xe.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2022

A stilbene–strontium iodide based radioxenon detection system for monitoring nuclear explosions

Atmospheric measurement of noble gases has been extensively used for monitoring clandestine nuclear weapon explosions for many years. The ratios of four xenon isotopes of interest ( 131 mXe, 133 mXe, 133 Xe, and 135 Xe) help in discriminating regular reactor operations from nuclear tests. A new coincidence-based detection system using stilbene and strontium iodide [SrI 2 (Eu)] for electron and photon detection respectively was developed at Oregon State University to address some of the challenges of the radioxenon systems deployed in the field such as memory effect, and poor energy resolution. Silicon photomultipliers (SiPMs) were used for sensing optical photons from all scintillation media. Real-time coincidence identification was achieved using the eight-channel digital pulse processor. The detection system was evaluated using lab check sources and Oregon State TRIGA reactor irradiated radioxenon samples. A 48-hour background coincidence spectrum was collected yielding a coincidence count rate and background rejection rate of 0.0174 ± 0.0003 counts per second (cps) and 98.9% respectively. The minimum detectable concentration (MDC) of the system was evaluated to be 0.11 ± 0.01, 0.13 ± 0.02, 0.20 ± 0.02, and 0.73 ± 0.08 for 131 mXe, 133 mXe, 133 Xe, and 135 Xe respectively. The memory effect of the detection system was found to be 0.069 ± 0.015%, which is almost a 70-fold reduction compared to traditional plastic scintillators. Here, the detection elements, custom-designed electronics, and the detector response to radioxenon are detailed in this work.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗
DOE OSTI2020

Scalable line and plane relaxation in a parallel structured multigrid solver

The efficient solution of sparse, linear systems that arise through the discretization of partial differential equations remains a key challenge for a range of high performance scientific simulations. One approach for reducing data movement and improving performance is by exposing and exploiting structure in a problem through the use of robust structured multilevel solvers. By choosing coarsening that preserves the structure of the problem, these methods maintain efficient structured computation and communication throughout the multigrid hierarchy. However, when coarsening is not permitted to be dependent on the operator, anisotropy must be addressed by the smoother — producing error compatible for coarse-grid correction with structured coarsening. Here, the components required in a scalable parallel structured solver are described with a focus on memory and communication efficiency of robust smoothers. While the implementation of communication and memory reduction techniques in smoothers integrated in a complete 3D solver present a significant engineering challenge, a novel approach is proposed that addresses these challenges systematically through a change to the solver’s execution model. Enabled by user-level threading paired with a set of data and communication abstractions, this approach permits seamless aggregation of communication in plane smoothers — directly reusing code for a 2D distributed multilevel cycle. Results show an effective reduction in communication costs for coarse-grid problems, and result in a speedup of 8.7x in smoothing routines shown in Fig. 12 using this approach. This produces a significant improvement to strong scalability while maintaining favorable weak scaling behavior. Finally, a parallel scaling study using a series of refined meshes is included that demonstrates the effectiveness of this approach in an application of interest.

97 MATHEMATICS AND COMPUTING↗