Engineering Papers⌕ Search

Engineering topics

Biondo, Elliott

Publications and source records attributed to Biondo, Elliott.

Celeritas: Accelerating Geant4 with GPUs

Celeritas [1] is a new Monte Carlo (MC) detector simulation code designed for computationally intensive applications (specifically, High Lumi- nosity Large Hadron Collider (HL-LHC) simulation) on high-performance heterogeneous architectures. In the past two years Celeritas has advanced from prototyping a GPU-based single physics model in infinite medium to implementing a full set of electromagnetic (EM) physics processes in complex geometries. The current release of Celeritas, version 0.3, has incorporated full device-based navigation, an event loop in the presence of magnetic fields, and detector hit scoring. New functionality incorporates a scheduler to offload electromagnetic physics to the GPU within a Geant4-driven simulation, enabling integration of Celeritas into high energy physics (HEP) experimental frameworks such as CMSSW. On the Summit supercomputer, Celeritas performs EM physics between 6 and 32 faster using the machine’s Nvidia GPUs compared to using only CPUs. When running a multithreaded Geant4 ATLAS test beam application with full hadronic physics, using Celeritas to accelerate the EM physics results in an overall simulation speedup of 1.8–2.3× on GPU and 1.2× on CPU.

Johnson, Seth R.↗

Celeritas R&D Report: Accelerating Geant4

Celeritas is a new Monte Carlo (MC) detector simulation code designed for computationally intensive applications on high-performance heterogeneous architectures. In the past two years Celeritas has advanced from prototyping a Graphics Processing Unit (GPU)-based single physics model in infinite medium to implementing a full set of electromagnetic (EM) physics processes in complex geometries. The current release of Celeritas, version 0.4, has incorporated full device-based navigation, an event loop in the presence of magnetic fields, and detector hit scoring. New functionality incorporates a scheduler to offload electromagnetic physics to the GPU within a Geant4-driven simulation, enabling straightforward integration of Celeritas into the high energy physics (HEP) experimental frameworks CMSSW and ATLAS FullSimLight. On the Perlmutter supercomputer, Celeritas performs EM physics between 3× and 18× faster using the machine’s Nvidia GPUs compared to using only CPUs, corresponding to an electrical power efficiency up to a factor of 5. When running a multithreaded Geant4 ATLAS test beam application with full hadronic physics, using Celeritas to accelerate the EM physics results in an overall simulation speedup of 1.7–2.2× on GPU and 1.2× on CPU. In a CMS test application using tt¯ events and the prototype Run 4 configuration, compared to Geant4 CPU, Celeritas with a Nvidia A100 improves overall throughput up to a factor of 2.7× but cannot be efficiently shared with more than 8 cores.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Exascale Multiphysics Nuclear Reactor Simulations for Advanced Designs

ENRICO is a coupled application developed under the U.S. Department of Energy's Exascale Computing Project (ECP) targeting the modeling of advanced nuclear reactors. It couples radiation transport with heat and fluid simulation, including the high-fidelity, highresolution Monte-Carlo code Shift and the Computational fluid dynamics code NekRS. NekRS is a highly-performant open-source code for simulation of incompressible and low-Mach fluid flow, heat transfer, and combustion with a particular focus on turbulent flows in complex domains. It is based on rapidly convergent high-order spectral element discretizations that feature minimal numerical dissipation and dispersion. State-of-the-art multilevel preconditioners, efficient high-order time-splitting methods, and runtime-adaptive communication strategies are built on a fast OCCA-based kernel library, libParanumal, to provide scalability and portability across the spectrum of current and future high-performance computing platforms. On Frontier, Nek5000/RS has recently achieved an unprecedented milestone in breaching over 1 billion spectral elements and 350 billion degrees of freedom. Shift has demonstrated the capability to transport upwards of 1 billion particles per second in full core nuclear reactor simulations featuring complete temperature-dependent, continuous-energy physics on Frontier. Shift achieved a weak-scaling efficiency of 97.8% on 8192 nodes of Frontier and calculated 6 reactions in 214,896 fuel pin regions below 1% statistical error yielding first-of-a-kind resolution for a Monte Carlo transport application.

Hamilton, Steven P.↗

Shutdown dose rate analysis with the Shift Monte Carlo radiation transport code and modular verification workflow

Calculation of the shutdown dose rate is crucial for safe fusion reactor operations. The Rigorous-two-step (R2S) method is a method that requires connected neutron transport, activation, and gamma transport. Shift has integrated variance reduction with a deterministic solver Denovo, supports multiple geometry formats, and is scalable. These features make it an attractive transport solver choice for an R2S workflow. An R2S workflow for the Shift Monte Carlo code is developed and compared to the existing Oak Ridge National Laboratory Shutdown Dose Rate Code Suite (ORCS) workflow. Also, a Python framework for integrating two R2S workflows is developed to mix and match each step in the R2S workflow for improved collaboration and verification experience. In this study, results show that the Shift-Denovo R2S workflow and the ORCS workflow calculate the shutdown dose rate of the ITER Shutdown Dose Rate benchmark problem with an average relative error of 2.285%.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Layered CAD/CSG geometry for spatially complex radiation transport scenarios

Many spatially complex fission, fusion, and national security Monte Carlo (MC) radiation transport scenarios involve combining computer-aided design (CAD) models with constructive solid geometry (CSG) models. A layered geometry method has been implemented in the Shift MC code to address this need. With layered geometry, multiple CAD and/or CSG models can be clipped, translated, rotated, and placed in overlapping layers to form transport-ready geometries. Here, the utility of this method is demonstrated with two problems: (1) a fixed-source simulation with a layered geometry consisting of a LiDAR-generated CAD model of the Combined Arms Collective Training Facility urban environment overlaid with CSG models of a mock hotel and a detector apparatus, and (2) a k-eigenvalue calculation using a layered geometry model of the Transformational Challenge Reactor consisting of CAD fuel elements placed in a CSG core. Tallied particle flux distributions match expectations, but tracking robustness must be improved prior to general-purpose use.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Singular value decomposition of adjoint flux distributions for Monte Carlo variance reduction

Monte Carlo (MC) shielding calculations often use weight windows (WWs) and biased sources formed from a deterministic estimate of the adjoint flux to improve the convergence rate of tallies. This requires a significant amount of computer memory, which can limit the memory available for high-resolution tally output. A new method is proposed for reducing these memory requirements by using singular value decomposition (SVD) in linear or logarithmic space to approximate the adjoint flux. This method’s performance is evaluated using the Shift and Denovo codes for streaming and diffusion base case problems, followed by problems using the Westinghouse AP1000 and the Joint European Torus. The log SVD reduced WW memory requirements by an order of magnitude in all cases without a significant performance penalty. Additionally, the linear SVD reduced biased source memory requirements by an order of magnitude, but further investigation is needed to account for observed limitations.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Rigorous Two-Step Activation for Fusion Systems with PyNE - Paper 127

In fusion systems, high energy neutrons activate system components. After shutdown, the biological dose rate from photons emitted from these components must be estimated as a function of position and time for maintenance planning. High resolution results can be obtained using the mesh-based Rigorous Two-Step (R2S) method. A neutron transport simulation provides mesh-based neutron fluxes. Using these fluxes and an appropriate irradiation scenario, a nuclear inventory analysis code is used to determine the photon source density within each mesh volume element at any decay time after shutdown. The photon source density mesh is used as a source for photon transport. Photon fluxes are tallied at positions of interest and flux-to-dose rate conversion factors are used to estimate the biological dose rate. Previous work by the authors involved the creation of R2S-ACT, an open source R2S workflow, using the Direct Accelerated Geometry Monte Carlo (DAGMC) version of MCNP5 (DAG-MCNP5) for both transport steps and the ALARA nuclear inventory analysis code. Various use cases may require different combinations of transport and activation codes, and nearly all of the capabilities of R2S-ACT have broader utility. For these reasons, the individual components of R2S-ACT were reimplemented within PyNE. PyNE is a collaborative, trans-institutional, open source project consisting of Python, Cython, and C++ code pertinent to nuclear simulations and analysis. This work involved additions to the PyNE mcnp, mesh, material, alara, dagmc, and sampling modules. A lightweight r2s module was then added to connect the code from these modules. The r2s and sampling modules, together with DAG-MCNP5 and ALARA form a complete R2S workflow for obtaining shutdown dose rate results. Mesh operations, geometry discretization, physics code input/output parsing, and mesh-based source sampling capabilities have been added to PyNE generically, eliminating the need to duplicate this effort across multiple applications. The PyNE r2s module uses these capabilities to couple MCNP and ALARA to form an R2S workflow, which has been validated with the FNG shutdown dose rate benchmark. Future work will involve benchmarking the workflow using tetrahedral mesh and also non-analog sampling options. added so that they can be used interchangeably. This will allow for comparisons between results obtained by different physics codes with minimal human effort. Work will also be done to create and implement an importance sampling method for biasing neutron transport towards regions of phase space of high importance to photon dose rate tallies. This will improve the efficiency of obtaining shutdown dose rate results in large, highly-attenuating geometries. (authors)

61 RADIATION PROTECTION AND DOSIMETRY↗