Engineering Papers⌕ Search

Engineering topics

Day, Marcus

Publications and source records attributed to Day, Marcus.

SUNDIALS time integrators for exascale applications with many independent systems of ordinary differential equations

Many complex systems can be accurately modeled as a set of coupled time-dependent partial differential equations (PDEs). However, solving such equations can be prohibitively expensive, easily taxing the world’s largest supercomputers. One pragmatic strategy for attacking such problems is to split the PDEs into components that can more easily be solved in isolation. This operator splitting approach is used ubiquitously across scientific domains, and in many cases leads to a set of ordinary differential equations (ODEs) that need to be solved as part of a larger “outer-loop” time-stepping approach. The SUNDIALS library provides a plethora of robust time integration algorithms for solving ODEs, and the U.S. Department of Energy Exascale Computing Project (ECP) has supported its extension to applications on exascale-capable computing hardware. In this paper, we highlight some SUNDIALS capabilities and its deployment in combustion and cosmology application codes (Pele and Nyx, respectively) where operator splitting gives rise to numerous, small ODE systems that must be solved concurrently.

97 MATHEMATICS AND COMPUTING↗

PeleMP: The Multiphysics Solver for the Combustion Pele Adaptive Mesh Refinement Code Suite

Combustion encompasses multiscale, multiphase reacting flow physics spanning a wide range of scales from the molecular scales, where chemical reactions occur, to the device scales, where the turbulent flow is affected by the geometry of the combustor. This scale disparity and the limited measurement capabilities from experiments make modeling combustion a significant challenge. Recent advancements in high-performance computing (HPC), particularly with the Department of Energy's Exascale Computing Project (ECP), have enabled high-fidelity simulations of practical applications to be performed. The major physics submodels, including chemical reactions, turbulence, sprays, soot, and thermal radiation, exhibit distinctive computational characteristics that need to be examined separately to ensure efficient utilization of computational resources. This paper presents the multiphysics solver for the Pele code suite, called PeleMP, which consists of models for spray, soot, and thermal radiation. Here, the mathematical and algorithmic aspects of the model implementations are described in detail as well as the verification process. The computational performance of these models is benchmarked on multiple supercomputers, including Frontier, an exascale machine. Results are presented from production simulations of a turbulent sooting ethylene flame and a bluff-body swirl stabilized spray flame with sustainable aviation fuels to demonstrate the capability of the Pele codes for modeling practical combustion problems with multiphysics. This work is an important step toward the exascale computing era for high-fidelity combustion simulations providing physical insights and data for predictive modeling of real-world devices.

42 ENGINEERING↗

Macrovoid resolved simulations of transport through HPRO relevant membrane geometries

Modeling the transport properties such as diffusivity and permeability of high pressure reverse osmosis (HPRO) membranes is critical for the selection and manufacture of membranes suitable for operation under high pressure. These properties can be significantly affected by the changes in heterogeneous pore structures due to compaction. Here, the modeling platform presented in this work resolves the two scale porosity in HPRO relevant membranes. A synthetic membrane geometry is constructed based on available experimental visualizations of pore structures. The simulations directly capture the flow channeling that results from a combination of material properties and geometric features of the macrovoids. A parametric study is presented to account for the transition of flow characteristics from a material governed regime to a macrovoid governed regime. Permeability of the membrane is evaluated using the simulation data and compared with an existing model that scales with the square of tortuosity over a range of material properties. The model is found to perform well under a narrow range of tortuosity while deviating from the calculated permeabilities at extreme conditions. The effective permeability of the membrane is found to vary by at least two orders of magnitude between the two flow regimes. It is also observed that tortuosity is a bounded property with its upper limit determined by the macrovoid geometry. Consequently, the tortuosity based correlations fail near a flow regime that is mainly governed by the macrovoids. The modeled permeability can be more than an order of magnitude smaller than the simulation result. A new model based on flux-weighted porosity of a membrane is introduced and its correlation with tortuosity is studied. The model agrees with the simulated data as, in addition to tortuosity, it also accounts for the flux partition within and outside the flow channels. Such correlations enable extending the existing understanding of flow characteristics to enhance predictability of porous media models.

42 ENGINEERING↗

PeleLMeX [SWR-22-48]

PeleLMeX is a solver for high fidelity reactive flow simulations, namely direct numerical simulation (DNS) and large eddy simulation (LES). The solver combines a low Mach number approach, adaptive mesh refinement (AMR), embedded boundary (EB) geometry treatment and high performance computing (HPC) to provide a flexible tool to address research questions on platforms ranging from small workstations to the world's largest GPU-accelerated supercomputers. PeleLMeX has been used to study complex flame/turbulence interactions in RCCI engines and hydrogen combustion or the effect of sustainable aviation fuel on gas turbine combustion. PeleLMeX is part of the Pele combustion Suite (https://amrex-combustion.github.io/)

Day, Marcus↗

Improving Gas Reactor Design with Complex Non-Standard Reaction Mechanisms in a Reactive Flow Model

Fluorinated gases are critical to a number of high-technology industries including semiconductor manufacturing and related processes such as flat panel display, LED, and photovoltaic manufacturing. These fluorinated gases have very high global warming potentials (GWPs) and, unfortunately are chemically very stable, in some cases lasting for thousands of years in the atmosphere. For environmental reasons, it is imperative that the release of these gases be minimized, but existing methods to destroy them requires a significant amount of energy. The goal of this project was to develop a more energy efficient method of destroying these high GWP gases using computational fluid dynamics, but the computational requirements are significant. In this project, we bring together resources available at the national labs, including high-performance computing hardware and modern computational methods, in order to explore an industrially important problem with key environmental and economic impact. In this project, we examined the state-of-the-art models for the physical processes involved, and made subtle changes to arrive at alternative formulations considerably better suited for computation and study. We examined several representative scenarios and developed a set of analyses techniques and tools that could be used to explore the systems in much greater detail. This exploratory project has demonstrated to ALZETA that HPC can indeed provide a unique view into their complex engineering problem - one that is simply inaccessible by any other means. The tools allow us to explore the details of the chemical process, and allow one to construct "what-if" scenarios to address shortcomings of the physical device in its current operating modes. However, as with many sources of completely new information, many new questions are illuminated with this capability. It will take considerable time, effort and experience before we learn to probe the solutions, pose operating scenarios and confirm modifications to the device that lead to real engineering advances.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

AMReX: a framework for block-structured adaptive mesh refinement

AMReX is a C++ software framework that supports the development of block-structured adaptive mesh refinement (AMR) algorithms for solving systems of partial differential equations (PDEs) with complex boundary conditions on current and emerging architectures. AMR reduces the computational cost and memory footprint compared to a uniform mesh while preserving the local descriptions of different physical processes in complex multiphysics algorithms. Current AMReX-based application codes span a number of areas, including atmospheric modeling, astrophysics, combustion, cosmology, fluctuating hydrodynamics, multiphase flows, and particle accelerators. In particular, the AMReX-Astro GitHub repository holds a number of astrophysical modeling tools based on AMReX. The origins of AMReX trace back to the BoxLib software framework. AMReX supports a number of different time-stepping strategies and spatial discretizations. Solution strategies supported by AMReX range from level-by-level approaches (with or without subcycling in time) with multilevel synchronization to full-hierarchy approaches, and any combination thereof. User-defined kernels that operate on patches of data can be written in C++ or Fortran; there is also a Fortran-interface functionality which wraps the core C++ data structures and operations in Fortran wrappers so that an application code based on AMReX can be written entirely in Fortran.

97 MATHEMATICS AND COMPUTING↗

A Bayesian approach to calibrating hydrogen flame kinetics using many experiments and parameters

First-principles Markov Chain Monte Carlo sampling is used to investigate uncertainty quantification and uncertainty propagation in parameters describing hydrogen kinetics. Specifically, we sample the posterior distribution for thirty-one parameters focusing on the H 2 O 2 and HO 2 reactions resulting from conditioning on ninety-one experiments. Established literature values are used for the remaining parameters in the mechanism as well as other thermodynamic and transport data needed to specify fluid properties. The samples are computed using an affine invariant sampler starting with broad, noninformative priors. Autocorrelation analysis shows that O(1M) samples are sufficient to obtain a reasonable sampling of the posterior. The resulting distribution identifies strong positive and negative correlations and several non-Gaussian characteristics. Using samples drawn from the posterior, we investigate the impact of parameter uncertainty on the prediction of two more complex flames: a 2D premixed flame kernel and the ignition of a hydrogen jet issuing into a heated chamber. The former represents a combustion regime similar to the target experiments used to calibrate the mechanism and the latter represents a different combustion regime. For the premixed flame, the net amount of product after a given time interval has a standard deviation of less than 2% whereas the standard deviation of the ignition time for the jet is more than 10%. The samples used for these studies are posted online. These results indicate the degree to which parameters consistent with the target experiments constrain predicted behavior in different combustion regimes. Finally, this process provides a framework for both identifying reactions for further study from candidate mechanisms as well as combining uncertainty quantification and propagation to, ultimately, tie uncertainty in laboratory flame experiments to uncertainty in end-use numerical predictions of more complicated scenarios.

97 MATHEMATICS AND COMPUTING↗

In situ and in-transit analysis of cosmological simulations

Abstract Modern cosmological simulations have reached the trillion-element scale, rendering data storage and subsequent analysis formidable tasks. To address this circumstance, we present a new MPI-parallel approach for analysis of simulation data while the simulation runs, as an alternative to the traditional workflow consisting of periodically saving large data sets to disk for subsequent ‘offline’ analysis. We demonstrate this approach in the compressible gasdynamics/ N -body code Nyx, a hybrid $\mbox{MPI}+\mbox{OpenMP}$ MPI + OpenMP code based on the BoxLib framework, used for large-scale cosmological simulations. We have enabled on-the-fly workflows in two different ways: one is a straightforward approach consisting of all MPI processes periodically halting the main simulation and analyzing each component of data that they own (‘ in situ ’). The other consists of partitioning processes into disjoint MPI groups, with one performing the simulation and periodically sending data to the other ‘sidecar’ group, which post-processes it while the simulation continues (‘in-transit’). The two groups execute their tasks asynchronously, stopping only to synchronize when a new set of simulation data needs to be analyzed. For both the in situ and in-transit approaches, we experiment with two different analysis suites with distinct performance behavior: one which finds dark matter halos in the simulation using merge trees to calculate the mass contained within iso-density contours, and another which calculates probability distribution functions and power spectra of various fields in the simulation. Both are common analysis tasks for cosmology, and both result in summary statistics significantly smaller than the original data set. We study the behavior of each type of analysis in each workflow in order to determine the optimal configuration for the different data analysis algorithms.

79 ASTRONOMY AND ASTROPHYSICS↗

Topology and burning rates of turbulent, lean, H 2 /air flames

Improved understanding of turbulent flames characterized by negative consumption speed-based Markstein lengths is necessary to develop better models for turbulent lean combustion of high hydrogen content fuels. In this paper we investigate the topology and burning rates of turbulent, lean (φ = 0.31), H 2 /air flames obtained from a recently published DNS database (Aspden et al., 2011). We calculate local flame front curvatures, strain rates, thicknesses, and burning velocities and compare these values to reference quantities obtained from stretched laminar flames computed numerically in three model geometrical configurations-a counterflow twin flame, a tubular counterflow flame and an expanding cylindrical flame. We compare and contrast the DNS with these model laminar flame calculations, and show both where they closely correlate with each other, as well as where they do not. These results in the latter case are shown to result from non-flamelet behaviors, unsteady effects, and curvature-strain correlations. These insights are derived from comparisons conditioned on different topological features, such as portions of the flame front with a spherical/cylindrical shape, the leading edge of the flame, and portions of the flame front with low mean curvature. We also show that reference time scales vary appreciably over the flame, and characterizing the relative values of fluid mechanic and kinetic time scales by a single value leads to erroneous conclusions. For example, there is a two order of magnitude decrease in chemical time scales at the leading edge of the front relative to its unstretched value. In conclusion, for this reason, the leading edge of the front quite closely tracks quasi-steady calculations, even in the lowest Damköhler number case, Da F ~0.005

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗