Engineering Papers⌕ Search

Engineering topics

Shephard, Mark S.

Publications and source records attributed to Shephard, Mark S..

At least 19 records

PCMS: Parallel Coupler For Multimodel Simulations

This paper presents the Parallel Coupler for Multimodel Simulations (PCMS), a new GPU accelerated generalized coupling framework for coupling simulation codes on leadership class supercomputers. PCMS includes distributed control and field mapping methods for up to five dimensions. For field mapping PCMS can utilize discretization and field information to accommodate physics constraints. PCMS is demonstrated with a coupling of the gyrokinetic microturbulence code XGC with a Monte Carlo neutral transport code DEGAS2 and with a 5D distribution function coupling of an energetic particle transport code (GNET) to a gyrokinetic microturbulence code (GTC). Weak scaling is also demonstrated on up to 2,080 GPUs of Frontier with a weak scaling efficiency of 85%.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Adaptive workflow for simulation of RF heaters

Accurate RF (Radio Frequency) simulations of fusion systems like ITER require the definition of high-fidelity analysis geometries that include detailed antenna, reactor wall, and physics regions. Here, this paper will describe a workflow for the execution of adaptive high-performance simulations of RF fusion systems. In this workflow, the simulation input consists of a CAD model attributed with the needed analysis attributes. The analysis mesh is automatically generated and the analysis steps are executed using the time-harmonic Maxwell's equations solved using high-order Nédélec finite elements. A patch recovery-based error estimator is used to drive a conforming mesh adaptation procedure.

97 MATHEMATICS AND COMPUTING↗

Port and optimize the CEED software stack to Aurora/Frontier EA (ECP Milestone Report)

The goal of this milestone was to port the CEED software stack, including Nek, MFEM and libCEED to the Frontier and Aurora early access hardware, work on optimizing the performance on AMD and Intel GPUs, and demonstrate impact in CEED-enabled ECP applications. As part of this milestone, we also performed a number of other activities, including continued optimizations for CEED applications at scale on Summit, performance evaluation on non-ECP hardware, including the Fugaku’s A64FX chip and the NVIDIA A100 GPU architecture, exploring the use of just-in-time compilations in applications at scale and potential of mixed precision optimizations in the CEED discretization and solver algorithms, and more. During the milestone period, the CEED team released new versions of 6 of its packages, including major releases of MFEM, NekRS and OCCA. We also organized the fifth CEED Annual meeting (CEED5AM) which included nearly 100 researchers from national labs, universities and industry.

97 MATHEMATICS AND COMPUTING↗

PUMIPic: A mesh-based approach to unstructured mesh Particle-In-Cell on GPUs

Unstructured mesh particle-in-cell, PIC, simulations executing on the current and next generation of massively parallel systems require new methods for both the mesh and particles to achieve performance and scalability on GPUs. The traditional approach to implementing PIC simulations defines data structures and algorithms in terms of particles with a full copy of the unstructured mesh on every process. To effectively scale the unstructured mesh and particles, mesh-based PIC uses the unstructured mesh as the predominant data structure with the particles stored in terms of the mesh entities. Here, this paper details the PUMIPic library, a framework for developing efficient and performance-portable mesh-based PIC simulations on GPU systems. A pseudo physics simulation based on a five-dimensional gyro-kinetic code for modeling plasma physics is used to examine the performance of PUMIPic. Scaling studies of the unstructured mesh partition and number of particles are performed up to 4096 nodes of the Summit system at Oak Ridge National Laboratory. The studies show that mesh-based PIC can utilize a partitioned mesh and maintain scaling up to system limitations.

97 MATHEMATICS AND COMPUTING↗

Efficient exascale discretizations: High-order finite element methods

Efficient exploitation of exascale architectures requires rethinking of the numerical algorithms used in many large-scale applications. These architectures favor algorithms that expose ultra fine-grain parallelism and maximize the ratio of floating point operations to energy intensive data movement. One of the few viable approaches to achieve high efficiency in the area of PDE discretizations on unstructured grids is to use matrix-free/partially assembled high-order finite element methods, since these methods can increase the accuracy and/or lower the computational time due to reduced data motion. In this paper we provide an overview of the research and development activities in the Center for Efficient Exascale Discretizations (CEED), a co-design center in the Exascale Computing Project that is focused on the development of next-generation discretization software and algorithms to enable a wide range of finite element applications to run efficiently on future hardware. CEED is a research partnership involving more than 30 computational scientists from two US national labs and five universities, including members of the Nek5000, MFEM, MAGMA and PETSc projects. We discuss the CEED co-design activities based on targeted benchmarks, miniapps and discretization libraries and our work on performance optimizations for large-scale GPU architectures. We also provide a broad overview of research and development activities in areas such as unstructured adaptive mesh refinement algorithms, matrix-free linear solvers, high-order data visualization, and list examples of collaborations with several ECP and external applications.

97 MATHEMATICS AND COMPUTING↗

The Exascale Framework for High Fidelity coupled Simulations (EFFIS): Enabling whole device modeling in fusion science

We present the Exascale Framework for High Fidelity coupled Simulations (EFFIS), a workflow and code coupling framework developed as part of the Whole Device Modeling Application (WDMApp) in the Exascale Computing Project. EFFIS consists of a library, command line utilities, and a collection of run-time daemons. Together, these software products enable users to easily compose and execute workflows that include: strong or weak coupling, in situ (or offline) analysis/visualization/monitoring, command-and-control actions, remote dashboard integration, and more. We describe WDMApp physics coupling cases and computer science requirements that motivate the design of the EFFIS framework. Furthermore, we explain the essential enabling technology that EFFIS leverages: ADIOS for performant data movement, PerfStubs/TAU for performance monitoring, and an advanced COUPLER for transforming coupling data from its native format to the representation needed by another application. Finally, we demonstrate EFFIS using coupled multi-simulation WDMApp workflows and exemplify how the framework supports the project’s needs. We show that EFFIS and its associated services for data movement, visualization, and performance collection does not introduce appreciable overhead to the WDMApp workflow and that the resource-dominant application’s idle time while waiting for data is minimal.

97 MATHEMATICS AND COMPUTING↗

High-order algorithmic developments and optimizations for large-scale GPU-accelerated simulations (Milestone CEED-MS36)

The goal of this milestone was to improve the high-order software ecosystem for CEED-enabled ECP applications by making progress on efficient matrix-free kernels targeting forthcoming ECP architectures. These kernels included matrix-free preconditioning and the development of new set of CEED solver bake-off problems. As part of this milestone, we also released the next version of the CEED software stack, CEED-4.0, reported on results from several application collaborations, and documented the efforts of porting to AMD GPUs for Frontier and other modern architectures, such as Fugaku. The specific tasks addressed in this milestone were: (1) Port and run CEED benchmarks/miniapps on Frontier EA systems; (2) Demonstrate performant libCEED integration in MFEM, Nek and applications; (3) Matrix-free preconditioning of high-order operators; (4) Benchmark problems for fast high-order solvers on GPU platforms; and (5) Public release of CEED-4.0. The artifacts delivered include the next version of the CEED software stack, CEED-4.0, the next libCEED release, libCEED-0.8, and a number of developments integrated within applications to improve their GPU and CPU performance and capabilities. See the CEED website, https://ceed.exascaleproject.org and the CEED GitHub organization, https://github.com/ceed for more details.

97 MATHEMATICS AND COMPUTING↗

Center for Extended Magnetohydrodynamic Modeling

The RPI team efforts focused on providing unstructured mesh technologies to the Center for Extended Magnetohydrodynamic Modeling’s efforts on the development of the M3D-C1 plasma physics code. Specific developments included: Providing the parallel unstructured mesh infrastructure used within the M3D-C1 code; Creating a tool for the generation and subsequent adaptation of the unstructured meshes; A set of procedures to be able to initiate an M3D-C1 simulation in 2D and automatically moving the simulation to a full 3D finite element model when 3D modes begin to appear; Developing a set of physics/mesh based diagonal preconditioners; Supporting the installation of M3D-C1 on various super computer systems; Supplementing the mesh generation procedures to support a model wall region to support the application of an advanced resistive wall model; Providing M3D-C1 with tools to support PIC operations on distributed meshes.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

In‐memory integration of existing software components for parallel adaptive unstructured mesh workflows

Summary Reliable mesh‐based simulations are needed to solve complex engineering problems. Mesh adaptivity can increase reliability by reducing discretization errors but requires multiple software components to exchange information. Often, components exchange information by reading and writing a common file format. This file‐based approach becomes a problem on massively parallel computers where filesystem bandwidth is a critical performance bottleneck. Our approach using data streams and component interfaces avoids the filesystem bottleneck. In this paper, we present these techniques and their use for coupling mesh adaptivity to the PHASTA computational fluid dynamics solver, the Albany multi‐physics framework, and the Omega3P linear accelerator frequency analysis applications. Performance results are reported on up to 16,384 cores of an Intel Knights Landing‐based system.

Smith, Cameron W.↗

Improving Unstructured Mesh Partitions for Multiple Criteria Using Mesh Adjacencies

The scalability of unstructured mesh based applications depends on partitioning methods that quickly balance the computational work while reducing communication costs. Zhou et al. [SIAM J. Sci. Comput., 32 (2010), pp. 3201{3227; J. Supercomput., 59 (2012), pp. 1218{1228] demonstrated the combination of (hyper)graph methods with vertex and element partition improvement for PHASTA CFD scaling to hundreds of thousands of processes. Our work generalizes partition improvement to support balancing combinations of all the mesh entity dimensions (vertices, edges, faces, regions) in partitions with imbalances exceeding 70%. Improvement results are then presented for multiple entity dimensions on up to one million processes on meshes with over 12 billion tetrahedral elements.

97 MATHEMATICS AND COMPUTING↗

Extreme Scale Unstructured Adaptive CFD: From Multiphase Flow to Aerodynamic Flow Control

Understanding the flow of fluid, either liquid or gas, through and around solid bodies has challenged man since the dawn of scientific inquiry. Many of the great minds of science and math have progressively built up a hierarchy of fluid models. This report is concerned with the computational modeling of turbulent flow around aerodynamic bodies such as planes and wind turbines. In this case, viscous effects near the solid bodies create very thin boundary layers that yield highly anisotropic (gradients normal to the surface may be 10^6 times larger than gradients along the surface) solutions to the governing non-linear partial differential equations (PDE); the Navier-Stokes equations. Furthermore, turbulent flows develop extremely broad ranges of length and time scales.This disparity motivates the use of discretization methods capable of employing adaptivity and implicit time integration. The combination of these features (non-linear, anisotropy, adaptivity, andimplicit) dramatically raise the complexity of the discretization, posing large challenges to efficient scalable parallel implementation. However, through careful design, the more complex algorithms can provide great reductions in computational cost relative to simpler methods (e.g., Cartesian grids with explicit time integration) that are easier to mate efficiently to hardware. In this report, we not only describe our approach but we also address the fact that while complex algorithms may never be as efficient flop-for-flop as simple methods, in the important measure of science-per-core-hour, they can still win big by making complex features like adaptivity and implicit methods as efficient and scalable as possible.

42 ENGINEERING↗

Modifiable Array Data Structures for Mesh Topology

Topological data structures are useful in many areas, including the various mesh data structures used in finite element and finite volume applications. Here, we present a mesh data structure which is both flexible enough to support general mesh adaptivity and is compactly stored in a few large arrays. This structure can efficiently store full topological information for meshes with mixed element types. It allows both constant-time adjacency queries and constant-time addition and removal of mesh entities. We compare this structure to similar structures for finite elements and graphics and present its performance for adjacency queries. We also present its use in adaptive simulation applications.

97 MATHEMATICS AND COMPUTING↗

Parallel unstructured grid generation for computational aerosciences

The objective of this research project is to develop efficient parallel automatic grid generation procedures for use in computational aerosciences. This effort is focused on a parallel version of the Finite Octree grid generator. Progress made during the first six months is reported.

Shephard, Mark S.↗

Thermoplastic matrix composites - Finite-element analysis of mode I and mode II failure specimens

A finite-element analysis was conducted to evaluate the stress distributions within mode I and mode II failure specimens, assuming both isotropic and orthotropic elastic material properties. The effects of anisotropy on both the magnitude and the location of the highest stress concentration at the vicinity of the crack tip are significant. The results from modeling realistic blunt crack tip geometry and resin-rich zones imply that local variations in the microstructure strongly influence the stress state near the crack tip and therefore the measured fracture properties. In addition, the features of a viscoelastic model for thermoplastic matrices are described. This model will be used in future investigations of matrix-dominated failure phenomena.

Bankert, Ray J.↗

Microstructural non-uniformity as related to unidirectional compression performance

The compression performance of unidirectional composite laminates (UDCLs) is related not only to intrinsic or local strength, but also to a sample's ability to resist propagaton of initial damage. This paper introduces a simple shear lag model which is applied to a compression loaded UDCL with initial damage. Among the information obtained are axial overload and matrix layer shear strain factors in the neighborhood of the damage zone. These factors indicate that propagation may occur with relatively little initial damage and emphasize the need for a more complete understanding of compression behavior than currently exists.

Majors, Patrick S.↗

Automatic crack growth tracking of bimaterial interface cracks

The propagation process of an interfacial crack in composite material is studied using the modified maximum dilatational strain energy density criterion, NT-criterion. Some necessary assumptions have been adopted to facilitate the use of the NT-criterion in this case. The stress intensity factors at the crack tip are extracted from the complex displacement field and finite element results. A simple algorithm for automatic crack propagation is presented with an illustrative example.

Yehia, Nabil A. B.↗

Integration of finite element modeling with solid modeling through a dynamic interface

Finite element modeling is dominated by geometric modeling type operations. Therefore, an effective interface to geometric modeling requires access to both the model and the modeling functionality used to create it. The use of a dynamic interface that addresses these needs through the use of boundary data structures and geometric operators is discussed.

Shephard, Mark S.↗