SEARCH · Engineering Papers
Results for “Mesh Adaptivity”
Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.
Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.
Emu v1.1
Emu is a particle-in-cell code for solving the neutrino quantum kinetic equations in 1, 2, or 3 spatial dimensions with arbitrary angular resolution in order to simulate neutrino flavor transformation in neutron star merger and core collapse supernova environments. Emu represents the neutrino distribution function as a set of particles, each of which represent a collection of neutrinos and antineutrinos with unique position and momentum. Each particle carries two density matrices to define the flavor state of the neutrinos and antineutrinos it represents. Emu includes the vacuum, matter, and neutrino self-interaction potentials. Emu calculates the self-interaction potential using PIC deposition and interpolation algorithms that efficiently compute the local number density and flux at particle locations. Emu is implemented in C++ and is based on the AMReX library for high-performance, block-structured adaptive mesh refinement. Emu is parallelized with MPI + OpenMP for CPUs and MPI + CUDA for GPUs.
PeleLMeX [SWR-22-48]
PeleLMeX is a solver for high fidelity reactive flow simulations, namely direct numerical simulation (DNS) and large eddy simulation (LES). The solver combines a low Mach number approach, adaptive mesh refinement (AMR), embedded boundary (EB) geometry treatment and high performance computing (HPC) to provide a flexible tool to address research questions on platforms ranging from small workstations to the world's largest GPU-accelerated supercomputers. PeleLMeX has been used to study complex flame/turbulence interactions in RCCI engines and hydrogen combustion or the effect of sustainable aviation fuel on gas turbine combustion. PeleLMeX is part of the Pele combustion Suite (https://amrex-combustion.github.io/)
mesoflow [SWR-22-56]
Mesoflow is a continuum scale simulation tool developed specifically for modeling transport and chemistry at the mesoscale. Our solver utilizes Cartesian block-structured adaptive mesh refinement to resolve complex surface morphologies (of catalysts/biomass particles among others) directly obtained from X-ray tomography data. An immersed boundary based formulation enables rapid representation of complex geometries prevalent in most mesoporous interfaces. The solver is developed on top of open-source performance portable library, AMReX, providing parallel execution capabilities on current and upcoming high-performance-computing (HPC) architectures. Our flexible software framework enables integration of complex chemical mechanisms at heterogenous interfaces and time-split algorithms for circumventing highly disparate reaction and flow time-scales. Our current studies indicate a ten-fold performance gain by using graphics-processing-units (GPU) compared to a single processor for representative problem sizes (2 million cell mesh).
PNNL-CompBio/BoltzmannMFX
BoltzmannMFX is a biological simulation code that solves chemical reaction networks using maximum entropy methods. It uses modules from MFiX-Exa and is based on the AMReX framework for massively parallel block-structured adaptive mesh applications.
HPGMG
HPGMG is a scalable reimplementation of miniGMG (see previous software disclosure) and is a compact geometric multigrid (MG benchmark designed to proxy the performance characteristics of the solves found in adaptive mesh refinement multigrid (AMR MG) applications. It solves the equation a*alpha*u - b div beta grad u = f on a large cubical domain using cell-centered values and either homogenous dirichlet or periodic boundaries. a and b are scalar constants, alpha and beta are space varying constants, f is the right hand side, and u is the solution. The cubical domain is divided into cubical subdomains which are distributed across the supercomputer. Whereas miniGMG enforced a static spatial decomposition across all levels in the MG V-cycle, HPGMG is free to redistribute data at each level including the possibility of moving all data to a single process (memory permitting). Thus, where miniGMG's U-cycle demanded a coarse grid solve across the entire supercomputer (a performance bottleneck), HPGMG can avoid this by combining distributed data and continuing the V-cycle. Technically, HPGMG can still implement a U-cycle or a Krylov bottom solve, but this solve can now be done on a global problem of only 2^3-8^3 cells. Additionally, HPGMG supports the MG F-cycle which avoids the iterative process of the v-cycle with a direct approach that can reach the discretization error in one pass. This provides a substantial performance boosts and opens avenues for exploiting systems that include hierarchical memories or heterogeneous processors. Like miniGMG, the righthand side f in HPGMG is generated from a continuous function for u. Thus, one can solve.
AMReX v2024
The software framework, AMReX, supports the development of block-structured adaptive mesh refinement (AMR) algorithms for solving systems of partial differential equations. AMR reduces the computational cost and memory footprint compared to a uniform mesh while preserving the essential local descriptions of different physical processes in complex multiphysics algorithms. AMR uses a hierarchical representation of the solution at multiple levels of resolution where the solution on each level is defined on the union of data containers at that resolution. These data containers, which represent the solution over a logically rectangular subregion of the domain, can contain field data defined on a mesh, Lagrangian particles or combinations of both. In addition to these basic data types, AMReX supports a multilevel embedded boundary representation of complex geometry; linear solvers for cell-centered and nodal data; asynchronous I/O in a native format readable by ParaView, VisIt and yt; and interfaces to hypre and PETSc solvers. AMReX enables applications to run on distributed memory architectures with multicore CPUs and with GPU accelerators. AMReX uses a lightweight abstraction layer that effectively hides the details of the architecture from the application. The framework currently supports CUDA, HIP and SYCL for GPU acceleration and OpenMP for multi-core CPU architectures.
CTH v.13.0
SAND2023-06647O The latest version of the CTH suite of computer codes is a flexible software system designed to treat a wide range of shock wave propagation and material motion phenomena in 1-D, 2-D, or 3-D. CTH has models suitable for most conditions encountered in shock physics including material strength, fracture, distended materials, high explosives, and a variety of boundary conditions. Finite-volume analogs of the Lagrangian equations of momentum and energy conservation are employed with continuous rezoning to construct Eulerian differencing. Adaptive mesh refinement is available for maximizing resolution in regions of interest. Full Lagrangian numerical methods are available for mechanical or structural response.
Vidyut3d: A Non-Equilibrium Plasma Modeling Tool [SWR-24-101]
Vidyut3d is a massively-parallel plasma-fluid solver for low-temperature plasmas (LTPs) that supports both local field (LFA) and local mean energy (LMEA) approximations, as well as complex gas and surface-phase chemistry. The solver supports 2D and 3D domains, and uses AMReX's adaptive mesh refinement capabilities to increase the grid resolution around complex structures (e.g. streamer heads and sheaths) while maintaining a tractable problem size. Vidyut specializes in simulating various types of gas-phase discharges, as well as plasma/surface interactions and surface chemistry (e.g. for plasma-mediated catalysis applications). The solver also supports hybrid CPU/GPU parallelization strategies, and has demonstrated excellent scaling on various HPC architectures for problem sizes consisting of O(100 M) control volumes.
CHEQUP v0.1
CHEQUP (Castro-based Hofi Expansion with QUasineutral Plasma) is a simulation code for modeling the formation of hydrodynamic optical-field-ionized (HOFI) plasma channels, which are used as waveguides in laser-plasma acceleration experiments. This includes experiments performed at LBNL's BELLA facility as well as other laser facilities across the world. CHEQUP extends the open-source Castro hydrodynamics framework with physics modules tailored for modeling HOFI plasma channels -- including multi-species ionization and three-body recombination for mixtures of hydrogen, nitrogen, helium, and argon ; a two-temperature model tracking electron and heavy-species temperatures separately ; and coupling with other codes of the BLAST ecosystem (https://blast.lbl.gov/) such as WarpX, via the openPMD standard. CHEQUP inherits from Castro the ability to run on modern GPU architectures (NVIDIA CUDA, AMD HIP) and supports adaptive mesh refinement (AMR) for efficient multi-scale resolution. Compared to existing tools, CHEQUP would be, to our knowledge, the first open-source code implementing the full HOFI channel formation physics, and the first implementation capable of running on GPUs. This enables significantly faster, large-scale parameter scans critical for the design of next-generation LPA-based accelerators and light sources.
Harnessing billions of tasks for a scalable portable hydrodynamic simulation of the merger of two stars
We propose a highly scalable demonstration of a portable asynchronous many-task programming model and runtime system applied to a grid-based adaptive mesh refinement hydrodynamic simulation of a double white dwarf merger with 14 levels of refinement that spans 17 orders of magnitude in astrophysical densities. The code uses the portable C++ parallel programming model that is embodied in the HPX library and being incorporated into the ISO C++ standard. The model reflects a significant shift from existing bulk synchronous parallel programming models under consideration for exascale systems. Through the use of the Futurization technique, seemingly sequential code is transformed into wait-free asynchronous tasks. We demonstrate the potential of our model by showing results from strong scaling runs on National Energy Research Scientific Computing Center’s Cori system (658,784 Intel Knight’s Landing cores) that achieve a parallel efficiency of 96.8% using billions of asynchronous tasks.
Efficient exascale discretizations: High-order finite element methods
Efficient exploitation of exascale architectures requires rethinking of the numerical algorithms used in many large-scale applications. These architectures favor algorithms that expose ultra fine-grain parallelism and maximize the ratio of floating point operations to energy intensive data movement. One of the few viable approaches to achieve high efficiency in the area of PDE discretizations on unstructured grids is to use matrix-free/partially assembled high-order finite element methods, since these methods can increase the accuracy and/or lower the computational time due to reduced data motion. In this paper we provide an overview of the research and development activities in the Center for Efficient Exascale Discretizations (CEED), a co-design center in the Exascale Computing Project that is focused on the development of next-generation discretization software and algorithms to enable a wide range of finite element applications to run efficiently on future hardware. CEED is a research partnership involving more than 30 computational scientists from two US national labs and five universities, including members of the Nek5000, MFEM, MAGMA and PETSc projects. We discuss the CEED co-design activities based on targeted benchmarks, miniapps and discretization libraries and our work on performance optimizations for large-scale GPU architectures. We also provide a broad overview of research and development activities in areas such as unstructured adaptive mesh refinement algorithms, matrix-free linear solvers, high-order data visualization, and list examples of collaborations with several ECP and external applications.
Towards exascale for wind energy simulations
We examine large-eddy-simulation modeling approaches and computational performance of two open-source computational fluid dynamics codes for the simulation of atmospheric boundary layer flows that are of direct relevance to wind energy production. The first code, NekRS, is a high-order, unstructured-grid, spectral element code. The second code, AMR-Wind, is a second-order, block-structured, finite-volume code with adaptive mesh refinement capabilities. The objective of this study is to co-develop these codes in order to improve model fidelity and performance for each. These features will be critical for running ABL-based applications such as wind farm analysis on advanced computing architectures. To this end, we investigate the performance of NekRS and AMR-Wind on the Oak Ridge Leadership Facility supercomputers Summit, using 4 to 800 nodes (24 to 4,800 NVIDIA V100 GPUs), and Crusher, the testbed for the Frontier exascale system, using 18 to 384 Graphics Compute Dies on AMD MI250X GPUs. We compare strong- and weak-scaling capabilities, linear solver performance, and time to solution. We also identify leading inhibitors to parallel scaling.
Computational study on the impact of gasoline-ethanol blending on autoignition and soot/NO x emissions under low-load gasoline compression ignition conditions
Here, in the present work, computational fluid dynamics (CFD) simulations of a single-cylinder gasoline compression ignition (GCI) engine are performed to investigate the impact of gasoline-ethanol blending on autoignition, nitrogen oxide (NO x ), and soot emissions under low-load conditions. In order to represent the test gasoline (RD5-87), a four-component toluene primary reference fuel (TPRF)+ethanol (ETPRF) surrogate (with 10% ethanol by volume; E10) is employed. A three-dimensional (3D) engine CFD model employing finite-rate chemistry with a skeletal kinetic mechanism (including NO x sub-mechanism), adaptive mesh refinement (AMR), and hybrid method of moments (HMOM) is adopted to capture the in-cylinder combustion phenomena and soot/NO x emissions. The engine CFD model is validated against experimental data for three gasoline-ethanol blends: E10, E30 and E100, with varying ethanol content by volume. Model validation is carried out for a broad range of start-of-injection (SOI) timings (−21, −27, −36, and −45 crank angle degrees (°CA) after top-dead-center (aTDC)) with respect to in-cylinder pressure, heat release rate, combustion phasing, NO x and soot emissions. For relatively later injection timings (−21 and −27 °CA aTDC), E30 yields higher amount of soot than E10; while the trend reverses for early injection cases (−36 and −45 °CA aTDC ). On the other hand, E100 yields the lowest amount of soot among all fuels irrespective of SOI timing. Further, E10 shows a non-monotonic trend in soot emissions with SOI timing: SOI-36>SOI-45>SOI-21>SOI-27, while soot emissions from E30 exhibit monotonic decrease with advancing SOI timing. NO x emissions from various fuels follow a trend of E10>E30>E100. On the other hand, NO x emissions increase as SOI timing is advanced for all fuels, with an anomaly for E10 and E100 where NO x decreases when SOI is advanced beyond −36 °CA aTDC. Detailed analysis of the numerical results is performed to investigate the soot/NO x emission trends and elucidate the impact of chemical composition and physical properties on autoignition and emissions characteristics.
Multiscale Modeling of Porous Medium Systems. In: Handbook of Porous Media, Third edition
The use of computers over the last 60 years for simulating subsurface “ow problems has provided an impressive, ever-expanding ability to model processes at a high resolution. Whereas in the 1960s a computational grid with 500 œxed nodes would have pushed the boundaries of computer power [59], simulations today involve millions of spatial grid points and adaptive meshes [40,57].
Hypersonic wake velocity measurements using acetone molecular tagging velocimetry
Molecular tagging velocimetry, a minimally intrusive laser diagnostic, was used to find quantitative off-body velocity measurements in the wake of both strut-mounted and free-flight spheres in hypersonic flow. Acetone gas seeded in the flow was excited using the fourth harmonic of a pulse-burst Nd:YAG laser. The experimental tests were validated and compared to simulations using a continuous Galerkin flow solver and an adaptive mesh refinement process. Near the centerline in the wake of the sphere, velocities between −105 and 65 m/s were observed due to reverse flow in the viscous shear layer. Similarly, the simulated data showed negative velocity values in the shear layer region. Outside of the shear layer, average velocities of 730 m/s were observed in both the experimental and simulated results. The overall average uncertainty for the strut-mounted and free-flight cases was estimated to have an accuracy of ±4% (±35.5m/s or 4.9%). In conclusion, this non-intrusive technique provides a velocity map of the wake behind a sphere under hypersonic conditions.
ERF: Energy Research and Forecasting
The Energy Research and Forecasting (ERF) code is a new model that simulates the mesoscale and microscale dynamics of the atmosphere using the latest high-performance computing architectures. It employs hierarchical parallelism using an MPI+X model, where X may be OpenMP on multicore CPU-only systems, or CUDA, HIP, or SYCL on GPU-accelerated systems. ERF is built on AMReX (Zhang et al., 2019, 2021), a block-structured adaptive mesh refinement (AMR) software framework that provides the underlying performance-portable software infrastructure for block-structured mesh operations. The "energy" aspect of ERF indicates that the software has been developed with renewable energy applications in mind. In addition to being a numerical weather prediction model, ERF is designed to provide a flexible computational framework for the exploration and investigation of different physics parameterizations and numerical strategies, and to characterize the flow field that impacts the ability of wind turbines to extract wind energy. The ERF development is part of a broader effort led by the US Department of Energy's Wind Energy Technologies Office.