Engineering Papers⌕ Search

Engineering topics

Slattery, Stuart

Publications and source records attributed to Slattery, Stuart.

At least 19 records

Co-design for Particle Applications at Exascale

Co-design across the Exascale Computing Project (ECP) has been critical for both enabling science applications and bringing disparate communities together. Developing and porting applications to the various high-performance computing (HPC) architectures on pre-exascale and exascale computers has been quite challenging due to the diversity of hardware features and software stacks. The Co-design Center for Particle Applications (CoPA) has developed and enhanced the Cabana and PROGRESS/BML libraries to facilitate the creation of new particle applications, make existing particle applications exascale capable, and allow teams to explore new capabilities. Particle methods from atomistic, mesoscale, continuum, through cosmological scales have been built with Cabana, along with new possibilities for application coupling. Similarly, the PROGRESS/BML library has enabled quantum particle applications with linear algebra solvers to use advanced hardware. Across these CoPA-developed libraries, the co-design abstraction layer combines performance portability with math library support to facilitate separation of concerns and directly support science runs.

97 MATHEMATICS AND COMPUTING↗

Electromagnetic Field Simulation Utilizing VERTEX

The understanding of electromagnetic fields is essential for many core applications at Oak Ridge National Laboratory (ORNL). Electromagnetic fields play an important role in particle accelerators, antennas, electric motors, power grids, and are principal part of nuclear fusion. We develop a numerical solver called able to treat the Maxwell system utilizing ORNL's VERTEX framework.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Cabana: A Performance Portable Library for Particle-Based Simulations

Particle-based simulations are ubiquitous throughout many fields of computational science and engineering, spanning the atomistic level with molecular dynamics (MD), to mesoscale particle-in-cell (PIC) simulations for solid mechanics, device-scale modeling with PIC methods for plasma physics, and massive N-body cosmology simulations of galaxy structures, with many other methods in between (Hockney & Eastwood, 1989). While these methods use particles to represent significantly different entities with completely different physical models, many low-level details are shared including performant algorithms for short- and/or long-range particle interactions, multi-node particle communication patterns, and other data management tasks such as particle sorting and neighbor list construction. Cabana is a performance portable library for particle-based simulations, developed as part of the Co-Design Center for Particle Applications (CoPA) within the Exascale Computing Project (ECP) (Alexander et al., 2020). The CoPA project and its full development scope, including ECP partner applications, algorithm development, and similar software libraries for quantum MD, is described in (Mniszewski et al., 2021). Cabana uses the Kokkos library for on-node parallelism (Edwards et al., 2014; Trott et al., 2022), enabling simulation on multi-core CPU and GPU architectures, and MPI for GPU-aware, multi-node communication. Cabana provides particle simulation capabilities on almost all current Kokkos backends, including serial execution, OpenMP (including OpenMP-Target for GPUs), CUDA (NVIDIA GPUs), HIP (AMD GPUs), and SYCL (Intel GPUs), providing a clear path for the coming generation of accelerator-based exascale hardware. Cabana builds on Kokkos by providing new particle data structures and particle algorithms resulting in a similar execution policy-based, node-level programming model that is intended to be used in addition to the core Kokkos library within an application. Cabana is designed as an application and physics agnostic, but particle-specific toolkit which can either be used to generate a new application, or to be used as needed in existing applications at various levels of invasiveness including through interfaces that wrap user memory in existing data structures.

97 MATHEMATICS AND COMPUTING↗

ExaAM: Metal additive manufacturing simulation at the fidelity of the microstructure

Additive manufacturing (AM), or 3D printing, of metals is transforming the fabrication of components, in part by dramatically expanding the design space, allowing optimization of shape and topology. However, although the physical processes involved in AM are similar to those of welding, a field with decades of experimental, modeling, simulation, and characterization experience, qualification of AM parts remains a challenge. The availability of exascale computational systems, particularly when combined with data-driven approaches such as machine learning, enables topology and shape optimization as well as accelerated qualification by providing process-aware, locally accurate microstructure and mechanical property models. We describe the physics components comprising the Exascale Additive Manufacturing simulation environment and report progress using highly resolved melt pool simulations to inform part-scale finite element thermomechanics simulations, drive microstructure evolution, and determine constitutive mechanical property relationships based on those microstructures using polycrystal plasticity. We report on implementation of these components for exascale computing architectures, as well as the multi-stage simulation workflow that provides a unique high-fidelity model of process–structure–property relationships for AM parts. In addition, we discuss verification and validation through collaboration with efforts such as AM-Bench, a set of benchmark test problems under development by a team led by the National Institute of Standards and Technology.

3D printing↗

Adaptive Spatially Aware I/O for Multiresolution Particle Data Layouts

Large-scale simulations on nonuniform particle distributions that evolve over time are widely used in cosmology, molecular dynamics, and engineering. Such data are often saved in an unstructured format that neither preserves spatial locality nor provides metadata for accelerating spatial or attribute subset queries, leading to poor performance of visualization tasks. Furthermore, the parallel I/O strategy used typically writes a file per process or a single shared file, neither of which is portable or scalable across different HPC systems. We present a portable technique for scalable, spatially aware adaptive aggregation that preserves spatial locality in the output. We evaluate our approach on two supercomputers, Stampede2 and Summit, and demonstrate that it outperforms prior approaches at scale, achieving up to 2.5× faster writes and reads for nonuniform distributions. Furthermore, the layout written by our method is directly suitable for visual analytics, supporting low-latency reads and attribute-based filtering with little overhead.

Usher, Will↗

picassodev/picasso

Picasso is a C++ template library for the development of exascale-class particle-in-cell methods. It contains data structures for particle and grid management using the ECP Kokkos and Cabana libraries, abstractions for particle and grid discretizations, and tools for interface tracking and reconstruction with particles.

Slattery, Stuart↗

A massively parallel and scalable multi-CPU material point method

Harnessing the power of modern multi-GPU architectures, we present a massively parallel simulation system based on the Material Point Method (MPM) for simulating physical behaviors of materials undergoing complex topological changes, self-collision, and large deformations. Our system makes three critical contributions. First, we introduce a new particle data structure that promotes coalesced memory access patterns on the GPU and eliminates the need for complex atomic operations on the memory hierarchy when writing particle data to the grid. Second, we propose a kernel fusion approach using a new Grid-to-Particles-to-Grid (G2P2G) scheme, which efficiently reduces GPU kernel launches, improves latency, and significantly reduces the amount of global memory needed to store particle data. Finally, we introduce optimized algorithmic designs that allow for efficient sparse grids in a shared memory context, enabling us to best utilize modern multi-GPU computational platforms for hybrid Lagrangian-Eulerian computational patterns. We demonstrate the effectiveness of our method with extensive benchmarks, evaluations, and dynamic simulations with elastoplasticity, granular media, and fluid dynamics. In comparisons against an open-source and heavily optimized CPU-based MPM codebase [Fang et al. 2019] on an elastic sphere colliding scene with particle counts ranging from 5 to 40 million, our GPU MPM achieves over 100x per-time-step speedup on a workstation with an Intel 8086K CPU and a single Quadro P6000 GPU, exposing exciting possibilities for future MPM simulations in computer graphics and computational science. Moreover, compared to the state-of-the-art GPU MPM method [Hu et al. 2019a], we not only achieve 2x acceleration on a single GPU but our kernel fusion strategy and Array-of-Structs-of-Array (AoSoA) data structure design also generalizes to multi-GPU systems. Our multi-GPU MPM exhibits near-perfect weak and strong scaling with 4 GPUs, enabling performant and large-scale simulations on a 10243 grid with close to 100 million particles with less than 4 minutes per frame on a single 4-GPU workstation and 134 million particles with less than 1 minute per frame on an 8-GPU workstation.

Wang, Xinlei↗

Algorithmic and GPU enhancements for molecular dynamics in Cabana and LAMMPS

The focus of this milestone was to implement and test new algorithms and optimizations for improved GPU performance by molecular dynamics (MD) and related particle codes. The work was done in the CoPA Cabana library, the CabanaMD mini-app which uses Cabana to implement and benchmark prototypical MD models, and the LAMMPS MD code which is used by the ECP EXAALT project as well as for materials modeling generally. Efforts included (a) implementation of new or optimized algorithms for fundamental MD kernels like neighbor-list building and FFTs which are also widely used in other particle methods, (b) performance testing of data structure layouts for both simple pairwise and computationally intense neural network potentials, (c) optimizations aimed at improving GPU performance for models with small atom counts, and (d) optimizations specific to the SNAP interatomic potential due to its importance for the EXAALT project.

36 MATERIALS SCIENCE↗

VERA Infrastructure

The Virtual Environment for Reactor Applications (VERA) Infrastructure provides the infrastructure to build, configure, test, install, and package for release the VERAS suite of codes that is being developed by the Consortium for the Advanced Simulation of Light-water reactors (CASL). It includes the list of codes that can be built within VERA, the location on the configuration-controlled CASL development computer, the version number used in any particular software installation, the location of the third-party libraries used for testing VERA on a suite of the CASL testing and development computers, and the scripts to configure, build, test, install, and package for release a particular version of VERA. The VERA Infrastructure contains a list of scripts, configuration options, and utilities that are specific to CASL, but contain no physics or numeric, nuclear technology, or encryption technology.

Baird, Mark↗

Development of Preliminary VERA-CS Crud Induced Localized Corrosion Modeling Capability

The Consortium for Advanced Simulation of Light Water Reactors (CASL) is developing multiphysics core-simulator software for light water reactors (LWRs) known as VERA-CS in order to improve the state of the art in modeling and simulation of challenge problems that are limiting to the nuclear industry. One of these challenge problems includes fuel rod crud deposition, which can lead to crud-induced power shift (CIPS) and crud-induced localized corrosion (CILC). This paper documents work that was performed to develop a preliminary CILC-modeling capability in VERA-CS in support of the crud challenge problem. The CILC capabilities were developed by coupling VERA-CS to the CASL-developed Cicada code, which provides 1D and 3D clad conduction and oxide growth modeling tools, as well as coupling to the CASL-developed MAMBA code, which is used for modeling clad crud deposition. An approach called rod thermal-hydraulic reconstruction (ROTHCON) was developed and integrated into VERA-CS. This allows the modeler to capture spacer-grid turbulence and heat transfer effects in the CTF subchannel code so that the spatial resolution of crud and oxide rod surface growth could be better resolved. After implementing these capabilities, several assessments were performed to ensure that the capabilities function as expected, and a pin-resolved quarter-core simulation was run as a demonstration.

Salko Jr, Robert↗

Development of Preliminary VERA-CS Crud Induced Localized Corrosion Modeling Capability

The objective of this milestone was to develop a crud-induced localized corrosion (CILC) screening capability in VERA-CS using existing codes and capabilities that had been developed in the Consortium for Advanced Simulation of Light Water Reactors (CASL) as part of the efforts to resolve the crud challenge problem. The milestone work focused on improving existing CASL software and implementing clad corrosion modeling into VERA-CS. Clad corrosion modeling was implemented by creating an application interface (API) in the Cicada software so that its clad heat transfer and corrosion models could be used in VERA-CS. Cicada was coupled to the subchannel code CTF for solving crud and corrosion growth. Additionally, a clad corrosion model was added directly to CTF to serve as an alternate approach. The rod thermal hydraulic reconstruction (ROTHCON) process, which is used for capturing computational fluid dynamics (CFD)-predicted rod-surface heat and turbulence behavior that can impact crud-induced power shift (CIPS) and CILC prediction, was also implemented into VERA-CS. Usability improvements were made to this feature, which was developed over several previous milestones, so that the VERA-CS user can link spacer grids in a VERA-CS model to an applicable pregenerated spacer grid data file if available. Several assessments were performed to demonstrate the proper functioning of the new capability. Sensitivity studies were performed to demonstrate the effect of crud lithium content, oxide thermal conductivity, Cicada mesh refinement, and ROTHCON coupling mesh refinement on predicted amounts of crud and oxide growth. A full-scale demonstration was performed by running Seabrook Cycle 5 with the CILC capability. Some general observations were made from the assessments. It was observed that predicted corrosion thickness is less than anticipated for CILC cases, so further work is needed to better understand and improve the corrosion modeling. Refining the coupling mesh in the ROTHCON process leads to increased corrosion and crud thickness, which can affect where crud and corrosion grow, how thick it gets, and how many rods experience crud and corrosion growth. Results indicate that applying the ROTHCON process and refining the coupling mesh leads to only a slight increase in total core maximum crud and corrosion thickness, but it leads to a significant increase in the number of rods with a maximum thickness that surpasses a set threshold value (i.e., more rods grow thicker corrosion and crud). Computational performance of the simulation is degraded by a factor of 5{6 with high coupling mesh refinement. The impact of the ROTHCON coupling mesh refinement on total core crud mass is less significant; however, using the ROTHCON process with CFD-generated data over the Yao{Hochreiter{Leech (YHL) model in CTF leads to significantly different amounts of total core crud mass. Future work on the CILC screening capability will include model improvement and validation, development and testing of a CILC-risk screening criteria, and performance improvements.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Local Improvement Results for Anderson Acceleration with Inaccurate Function Evaluations

Here, we analyze the convergence of Anderson acceleration when the fixed point map is corrupted with errors. We also consider uniformly bounded errors and stochastic errors with infinite tails. We prove local improvement results which describe the performance of the iteration up to the point where the accuracy of the function evaluation causes the iteration to stagnate. We illustrate the results with examples from neutronics.

97 MATHEMATICS AND COMPUTING↗

Warthog: A MOOSE-Based Application for the Direct Code Coupling of BISON and PROTEUS

The Nuclear Energy Advanced Modeling and Simulation (NEAMS) program from the Department of Energy's Office of Nuclear Energy provides a robust toolkit for the modeling and simulation of current and future advanced nuclear reactor designs. This toolkit provides these technologies organized across product lines: two divisions targeted at fuels and end-to-end reactor modeling, and a third for integration, coupling, and high-level workflow management. The Fuels Product Line and the Reactor Product line provide advanced computational technologies that serve each respective field well, however, their current lack of integration presents a major impediment to future improvements of simulation solution fidelity. There is a desire for the capability to mix and match tools across Product Lines in an effort to utilize the best from both to improve NEAMS modeling and simulation technologies. This report details a new effort to provide this Product Line interoperability through the development of a new application called Warthog. This application couples the BISON Fuel Performance application from the Fuels Product Line and the PROTEUS Core Neutronics application from the Reactors Product Line in an effort to utilize the best from all parts of the NEAMS toolkit and improve overall solution fidelity of nuclear fuel simulations. To achieve this, Warthog leverages as much prior work from the NEAMS program as possible, and in doing so, enables interoperability between the disparate MOOSE and SHARP frameworks, and the libMesh and MOAB mesh data formats. This report describes this work in full. We begin with a detailed look at the individual NEAMS framework technologies used and developed in the various Product Lines, and the current status of their interoperability. We then introduce the Warthog application: its overall architecture and the ways it leverages the best existing tools from across the NEAMS toolkit to enable BISON-PROTEUS integration. Furthermore, we show how Warthog leverages a tool known as DataTransferKit to seamlessly enable the transfer for solution data between disparate frameworks and mesh formats. To end, we demonstrate tests for the direct software coupling of BISON and PROTEUS using Warthog, and discuss current impediments and solutions to the construction of physically realistic input models for this coupled BISON-PROTEUS system.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

A Monte Carlo synthetic-acceleration method for solving the thermal radiation diffusion equation

We present a novel synthetic-acceleration-based Monte Carlo method for solving the equilibrium thermal radiation diffusion equation in three spatial dimensions. The algorithm performance is compared against traditional solution techniques using a Marshak benchmark problem and a more complex multiple material problem. Our results show that our Monte Carlo method is an effective solver for sparse matrix systems. For solutions converged to the same tolerance, it performs competitively with deterministic methods including preconditioned conjugate gradient and GMRES. We also discuss various aspects of preconditioning the method and its general applicability to broader classes of problems.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗