Engineering Papers⌕ Search

Engineering topics

Woodward, Carol S.

Publications and source records attributed to Woodward, Carol S..

At least 19 records

SUNDIALS time integrators for exascale applications with many independent systems of ordinary differential equations

Many complex systems can be accurately modeled as a set of coupled time-dependent partial differential equations (PDEs). However, solving such equations can be prohibitively expensive, easily taxing the world’s largest supercomputers. One pragmatic strategy for attacking such problems is to split the PDEs into components that can more easily be solved in isolation. This operator splitting approach is used ubiquitously across scientific domains, and in many cases leads to a set of ordinary differential equations (ODEs) that need to be solved as part of a larger “outer-loop” time-stepping approach. The SUNDIALS library provides a plethora of robust time integration algorithms for solving ODEs, and the U.S. Department of Energy Exascale Computing Project (ECP) has supported its extension to applications on exascale-capable computing hardware. In this paper, we highlight some SUNDIALS capabilities and its deployment in combustion and cosmology application codes (Pele and Nyx, respectively) where operator splitting gives rise to numerous, small ODE systems that must be solved concurrently.

97 MATHEMATICS AND COMPUTING↗

Performance of explicit and IMEX MRI multirate methods on complex reactive flow problems within modern parallel adaptive structured grid frameworks

Large-scale multiphysics simulations are computationally challenging due to the coupling of multiple processes with widely disparate time scales. The advent of exascale computing systems exacerbates these challenges since these systems enable ever-increasing size and complexity. In recent years, there has been renewed interest in developing multirate methods as a means to handle the large range of time scales, as these methods may afford greater accuracy and efficiency than more traditional approaches of using implicit-explicit (IMEX) and low-order operator splitting schemes. However, to date there have been few performance studies that compare different classes of multirate integrators on complex application problems. In this work, we study the performance of several newly developed multirate infinitesimal (MRI) methods, implemented in the SUNDIALS solver package, on two reacting flow model problems built on structured mesh frameworks. The first model revisits prior work on a compressible reacting flow problem with complex chemistry that is implemented using BoxLib but where we now include comparisons between a new explicit MRI scheme with the multirate spectral deferred correction (SDC) methods in the original paper. The second problem uses the same complex chemistry as the first problem, combined with a simplified flow model, but runs at a large spatial scale where explicit methods become infeasible due to stability constraints. Two recently developed IMEX MRI multirate methods are tested. These methods rely on advanced features of the AMReX framework on which the model is built, such as multilevel grids and multilevel preconditioners. The results from these two problems show that MRI multirate methods can offer significant performance benefits on complex multiphysics application problems and that these methods may be combined with advanced spatial discretization to compound the advantages of both.

97 MATHEMATICS AND COMPUTING↗

ARKODE: A Flexible IVP Solver Infrastructure for One-step Methods

We describe the ARKODE library of one-step time integration methods for ordinary differential equation (ODE) initial-value problems (IVPs). In addition to providing standard explicit and diagonally implicit Runge–Kutta methods, ARKODE supports one-step methods designed to treat additive splittings of the IVP, including implicit-explicit (ImEx) additive Runge–Kutta methods and multirate infinitesimal (MRI) methods. We present the role of ARKODE within the SUNDIALS suite of time integration and nonlinear solver libraries, the core ARKODE infrastructure for utilities common to large classes of one-step methods, as well as its use of “time stepper” modules enabling easy incorporation of novel algorithms into the library. Numerical results show example problems of increasing complexity, highlighting the algorithmic flexibility afforded through this infrastructure, and include a larger multiphysics application leveraging multiple algorithmic features from ARKODE and SUNDIALS.

97 MATHEMATICS AND COMPUTING↗

Removing Numerical Pathologies in a Turbulence Parameterization Through Convergence Testing

Abstract Discretized numerical models of the atmosphere are usually intended to faithfully represent an underlying set of continuous equations, but this necessary condition is violated sometimes by subtle pathologies that have crept into the discretized equations. Such pathologies can introduce undesirable artifacts, such as sawtooth noise, into the model solutions. The presence of these pathologies can be detected by numerical convergence testing. This study employs convergence testing to verify the discretization of the Cloud Layers Unified By Binormals (CLUBB) model of clouds and turbulence. That convergence testing identifies two aspects of CLUBB's equation set that contribute to undesirable noise in the solutions. First, numerical limiters (i.e., clipping) used by CLUBB introduce discontinuities or slope discontinuities in model fields. Second, nonlinear artificial diffusion employed for improving numerical stability can introduce unintended small‐scale features into the solution of the model equations. Smoothing the limiters and using linear artificial diffusion reduces the noise and restores the expected first‐order convergence in CLUBB's solutions. These model reformulations enhance our confidence in the trustworthiness of solutions from CLUBB by eliminating the unphysical oscillations in high‐resolution simulations. The improvements in the results at coarser, near‐operational grid spacing and timestep are also seen in cumulus cloud and dry turbulence tests. In addition, convergence testing is proven to be a valuable tool for detecting pathologies, including unintended discontinuities and grid dependence, in the model equation set.

58 GEOSCIENCES↗

Enabling New Flexibility in the SUNDIALS Suite of Nonlinear and Differential/Algebraic Equation Solvers

In recent years, the SUite of Nonlinear and DIfferential/ALgebraic equation Solvers (SUNDIALS) has been redesigned to better enable the use of application-specific and third-party algebraic solvers and data structures. Throughout this work, we have adhered to specific guiding principles that minimized the impact to current users while providing maximum flexibility for later evolution of solvers and data structures. The redesign was done through the addition of new linear and nonlinear solvers classes, enhancements to the vector class, and the creation of modern Fortran interfaces. The vast majority of this work has been performed “behind-the-scenes,” with minimal changes to the user interface and no reduction in solver capabilities or performance. These changes allow SUNDIALS users to more easily utilize external solver libraries and create highly customized solvers, enabling greater flexibility on extreme-scale, heterogeneous computational architectures.

97 MATHEMATICS AND COMPUTING↗

Removing numerical pathologies in a turbulence parameterization through convergence testing

Discretized numerical models of the atmosphere are usually intended to faithfully represent an underlying set of continuous equations, but this necessary condition is violated sometimes by subtle pathologies that have crept into the discretized equations. Such pathologies can introduce undesirable artifacts, such as sawtooth noise, into the model solutions. The presence of these pathologies can be detected by numerical convergence testing. This study employs convergence testing to verify the discretization of the Cloud Layers Unified By Binormals (CLUBB) model of clouds and turbulence. That convergence testing identifies two aspects of CLUBB's equation set that contribute to undesirable noise in the solutions. First, numerical limiters (i.e. clipping) used by CLUBB introduce discontinuities or slope discontinuities in model fields. Second, this noise can be amplified by an advective term in CLUBB's background diffusion. Smoothing the limiters and removing the advective component of the background diffusion reduces the noise and restores the expected first-order convergence in CLUBB's solutions. These model reformulations improve the results at coarser, near-operational grid spacing and time step in cumulus cloud and dry turbulence tests. In addition, convergence testing is proved to be a valuable tool for detecting pathologies, including unintended discontinuities and grid dependence, in the model equation set.

Mathematics and Computing, Geosciences↗

New efforts in improving numerical aspects of atmospheric physics parameterizations

This is an invited article for GEWEX Quarterly, the newsletter of the Global Energy and Water Exchanges (GEWEX) program which is a core project of the World Climate Research Programme (https://www.gewex.org/resources/gewex-news/). The article introduces recent research results and ongoing activities being carried out in a SciDAC project funded by DOE that focuses on identifying and addressing numerical issues in physics parameterizations in global atmospheric general circulation models.

global atmospheric model↗

Implicit Multirate GARK Methods

This article considers multirate generalized-structure additively partitioned Runge–Kutta methods for solving stiff systems of ordinary differential equations with multiple time scales. These methods treat different partitions of the system with different timesteps for a more targeted and efficient solution compared to monolithic single rate approaches. With implicit methods used across all partitions, methods must find a balance between stability and the cost of solving nonlinear equations for the stages. In order to characterize this important trade-off, we explore multirate coupling strategies, problems for assessing linear stability, and techniques to efficiently implement Newton iterations for stage equations. Unlike much of the existing multirate stability analysis which is limited in scope to particular methods, we present general statements on stability and describe fundamental limitations for certain types of multirate schemes. New implicit multirate methods up to fourth order are derived, and their accuracy and efficiency properties are verified with numerical tests.

97 MATHEMATICS AND COMPUTING↗

Data for publication Wan et al. (2020, JAMES), Improving time-step convergence in an atmosphere model with simplified physics: the impacts of closure assumption and process coupling

This dataset contains simple condensation mdoel output and CAM4 model output that were used to produce figures in the following paper: Wan, H., Woodward, C. S., Zhang, S., Vogl, C. J., Stinis, P., Gardner, D. J., Rasch, P. J., Zeng, X., Larson, V. E., and Singh, B. (2020): Improving time-step convergence in an atmosphere model with simplified physics: the impacts of closure assumption and process coupling, Journal of Advances in Modeling Earth Systems, https://doi.org/10.1029/2019MS001982

54 ENVIRONMENTAL SCIENCES↗

Simulating coupled surface–subsurface flows with ParFlow v3.5.0: capabilities, applications, and ongoing development of an open-source, massively parallel, integrated hydrologic model

Surface flow and subsurface flow constitute a naturally linked hydrologic continuum that has not traditionally been simulated in an integrated fashion. Recognizing the interactions between these systems has encouraged the development of integrated hydrologic models (IHMs) capable of treating surface and subsurface systems as a single integrated resource. IHMs are dynamically evolving with improvements in technology, and the extent of their current capabilities are often only known to the developers and not general users. This article provides an overview of the core functionality, capability, applications, and ongoing development of one open-source IHM, ParFlow. ParFlow is a parallel, integrated, hydrologic model that simulates surface and subsurface flows. ParFlow solves the Richards equation for three-dimensional variably saturated groundwater flow and the two-dimensional kinematic wave approximation of the shallow water equations for overland flow. The model employs a conservative centered finite-difference scheme and a conservative finite-volume method for subsurface flow and transport, respectively. ParFlow uses multigrid-preconditioned Krylov and Newton–Krylov methods to solve the linear and nonlinear systems within each time step of the flow simulations. The code has demonstrated very efficient parallel solution capabilities. ParFlow has been coupled to geochemical reaction, land surface (e.g., the Common Land Model), and atmospheric models to study the interactions among the subsurface, land surface, and atmosphere systems across different spatial scales. This overview focuses on the current capabilities of the code, the core simulation engine, and the primary couplings of the subsurface model to other codes, taking a high-level perspective.

58 GEOSCIENCES↗

Data from Wan et al. (2020, JAMES, DOI: 10.1029/2019MS001982)

This data set includes the simple condensation model output and CAM4 model output used to generate figures presented in the following paper: Wan, H., Woodward, C. S., Zhang, S., Vogl, C. J., Stinis, P., Gardner, D. J., Rasch, P. J., Zeng, X., Larson, V. E., and Singh, B. (2020): Improving time-step convergence in an atmosphere model with simplified physics: the impacts of closure assumption and process coupling, Journal of Advances in Modeling Earth Systems, https://doi.org/10.1029/2019MS001982

54 ENVIRONMENTAL SCIENCES↗

Evaluation of Implicit‐Explicit Additive Runge‐Kutta Integrators for the HOMME‐NH Dynamical Core

Abstract The nonhydrostatic High‐Order Method Modeling Environment (HOMME‐NH) atmospheric dynamical core supports acoustic waves that propagate significantly faster than the advective wind speed, thus greatly limiting the time step size that can be used with standard explicit time integration methods. Resolving acoustic waves is unnecessary for accurate climate and weather prediction. This numerical stiffness is addressed herein by considering implicit‐explicit additive Runge‐Kutta (ARK IMEX) methods that can treat the acoustic waves in a stable manner without requiring implicit treatment of nonstiff modes. Various ARK IMEX methods are evaluated for their efficiency in producing accurate solutions, ability to take large time step sizes, and sensitivity to grid cell length ratio. Both the gravity wave test and baroclinic instability test from the 2012 Dynamical Core Model Intercomparison Project are used to recommend 5 of the 27 ARK IMEX methods tested for use in HOMME‐NH.

97 MATHEMATICS AND COMPUTING↗

A parallel-in-time algorithm for variable step multistep methods

This paper presents a multigrid reduction in time (MGRIT) algorithm for achieving time parallelism using multistep backward difference formula (BDF) methods on variably-spaced temporal grids. This MGRIT approach transforms the linear multistep methods into single step methods applied to groups of time steps. Stability considerations are addressed through lowering of the order on coarse grids. The methods are presented for fixed and variable time step formulations. Furthermore, numerical results show moderate speedups for the heat equation as well as two IEEE power grid test problems characterized by nonlinear differential algebraic equations (DAE) of index 1. The performance of MGRIT with BDF is compared to MGRIT with Runge-Kutta for both the fixed and variable step methods of the same order.

97 MATHEMATICS AND COMPUTING↗

SUNDIALS Multiphysics+MPIManyVector Performance Testing

In this report we document performance test results on a SUNDIALS-based multiphysics demonstration application. We aim to assess the large-scale parallel performance of new capabilities that have been added to the SUNDIALS suite of time integrators and nonlinear solvers in recent years under funding from both the Exascale Computing Project (ECP) and the Scientific Discovery through Advanced Scientific (SciDAC) program, specifically: 1. SUNDIALS’ new MPIManyVector module, that allows extreme flexibility in how a solution "vector" is staged on computational resources. 2. ARKode’s new multirate integration module, MRIStep, allowing high-order accurate calculations that subcycle "fast" processes within "slow" ones. 3. SUNDIALS’ new flexible linear solver interfaces, that allow streamlined specification of problem-specific linear solvers. 4. SUNDIALS’ new N_Vector additions of "fused" vector operations (to increase arithmetic intensity) and separation of reduction operations into "local" and "global" versions (to reduce latency by combining multiple reductions into a single MPI_Allreduce call). We anticipate that subsequent reports will extend this work to investigate a variety of other new features, including SUNDIALS’ generic SUNNonlinearSolver interface and accelerator-enabled N Vector modules, and upcoming MRIStep extensions to support custom "fast" integrators (that leverage problem structure) and IMEX integration of the "slow" time scale (to add diffusion).

97 MATHEMATICS AND COMPUTING↗

An Objective and Efficient Method for Assessing the Impact of Reduced-Precision Calculations On Solution Correctness

Recent studies have shown that reducing the precision of floating-point calculations in an atmospheric model can improve the models' computational performance without affecting model fidelity, but code changes are needed to accommodate lower precision, and a small number of calculations might need to retain higher precision to prevent undue round-off error. For complex and computationally expensive systems like the Energy Exascale Earth System Model (E3SM), a method is needed to objectively assess the quality of the lower-precision simulations and to quickly identify problematic code pieces This paper demonstrates that solution correctness can be evaluated through a simple and quantitative error metric based on time step convergence. The proposed test method can unambiguously detect the accumulation of precision error and objectively assess its impact on solution accuracy. Since fast physical processes are known to affect key features of the multi-year mean climate in an atmospheric model, we show that the convergence test applied to short simulations can provide useful information about the impact of reduced precision on a model’s long-term behavior. In contrast, the traditional way of climate model evaluation requires multi-year simulations and involves inspecting many physical quantities for statistical significance in the presence of natural variability. The simplicity, computational efficiency, and objectivity of the proposed method makes it very attractive for high-resolution model development. The method is also expected to be applicable to other models that numerically solve time evolution equations.

97 MATHEMATICS AND COMPUTING↗

Performance analysis of fully explicit and fully implicit solvers within a spectral element shallow-water atmosphere model

In this study, explicit Runge–Kutta methods and implicit multistep methods utilizing a Newton–Krylov nonlinear solver are evaluated for a range of configurations of the shallow-water dynamical core of the spectral element community atmosphere model to evaluate their computational performance. These configurations are designed to explore the attributes of each method under different but relevant model usage scenarios including varied spectral order within an element, static regional refinement, and scaling to the largest problem sizes. This analysis is performed within the shallow-water dynamical core option of a full climate model code base to enable a wealth of simulations for study, with the aim of informing solver development within the more complete hydrostatic dynamical core used for climate research. The limitations and benefits to using explicit versus implicit methods, with different parameters and settings, are discussed in light of the trade-offs with Message Passing Interface (MPI) communication and memory and their inherent efficiency bottlenecks. Given the performance behavior across the configurations analyzed here, the recommendation for future work using the implicit solvers is conditional based on scale separation and the stiffness of the problem. For the regionally refined configurations, the implicit method has about the same efficiency as the explicit method, without considering efficiency gains from a preconditioner. The potential for improvement using a preconditioner is greatest for higher spectral order configurations, where more work is shifted to the linear solver. Finally, initial simulations with OpenACC directives to utilize a Graphics Processing Unit (GPU) when performing function evaluations show improvements locally, and that overall gains are possible with adjustments to data exchanges.

58 GEOSCIENCES↗

On the use of finite difference matrix-vector products in Newton-Krylov solvers for implicit climate dynamics with spectral elements

Efficient solutions of global climate models require effectively handling disparate length and time scales. Implicit solution approaches allow time integration of the physical system with a step size governed by accuracy of the processes of interest rather than by stability of the fastest time scales present. Implicit approaches, however, require the solution of nonlinear systems within each time step. Usually, a Newton's method is applied to solve these systems. Each iteration of the Newton's method, in turn, requires the solution of a linear model of the nonlinear system. This model employs the Jacobian of the problem-defining nonlinear residual, but this Jacobian can be costly to form. If a Krylov linear solver is used for the solution of the linear system, the action of the Jacobian matrix on a given vector is required. In the case of spectral element methods, the Jacobian is not calculated but only implemented through matrix-vector products. The matrix-vector multiply can also be approximated by a finite difference approximation which may introduce inaccuracy in the overall nonlinear solver. In this paper, we review the advantages and disadvantages of finite difference approximations of these matrix-vector products for climate dynamics within the spectral element shallow water dynamical core of the Community Atmosphere Model.

54 ENVIRONMENTAL SCIENCES↗