Engineering Papers⌕ Search

Engineering topics

Min, Misun

Publications and source records attributed to Min, Misun.

At least 19 records

Accelerating transients with NekRS: GPU overlapping domain implementation and multi-rate timestepping

The simulation of nuclear transients using Computational Fluid Dynamics (CFD) presents significant computational challenges due to the inherent complexity and the wide separation in temporal scales between various flow physical phenomena. These disparities lead to high computational costs, often making the simulation of transients impractical without advanced techniques. Consequently, multiple research initiatives are being pursued by the NEAMS thermal-hydraulic area, some driven by academic institutions and some by national laboratories. Overall, they are exploring novel methods to make transient simulations more feasible and efficient. This report delves into recent advancements within the CFD code NekRS, specifically those achieved in Fiscal Year 2024 under the CONNECT effort, aimed at improving the performance and feasibility of transient simulations. The first major advancement involves the porting of NekRS to Aurora, one of the Department of Energy’s (DOE) most powerful supercomputers. Additionally, the report discusses the implementation of an overlapping domain capability within NekRS. This novel GPU-accelerated capability allows different spatial regions of the domain to be solved independently, enhancing the code’s efficiency, particularly when running large-scale simulations in complex domains. The scalability of this approach is demonstrated, highlighting its potential to transform how transients are approached in CFD simulations. Lastly, the report focuses on how this overlapping domain capability specifically accelerates transient simulations through multi-rate timestepping. By decoupling different regions and facilitating faster computations, this method offers a promising pathway to making nuclear transient simulations more computationally feasible, addressing one of the critical bottlenecks in the field. Together, these advancements represent a significant leap forward in transient simulation technology, bringing closer the possibility of handling highly complex nuclear scenarios with greater efficiency and accuracy.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Towards exascale for wind energy simulations

We examine large-eddy-simulation modeling approaches and computational performance of two open-source computational fluid dynamics codes for the simulation of atmospheric boundary layer flows that are of direct relevance to wind energy production. The first code, NekRS, is a high-order, unstructured-grid, spectral element code. The second code, AMR-Wind, is a second-order, block-structured, finite-volume code with adaptive mesh refinement capabilities. The objective of this study is to co-develop these codes in order to improve model fidelity and performance for each. These features will be critical for running ABL-based applications such as wind farm analysis on advanced computing architectures. To this end, we investigate the performance of NekRS and AMR-Wind on the Oak Ridge Leadership Facility supercomputers Summit, using 4 to 800 nodes (24 to 4,800 NVIDIA V100 GPUs), and Crusher, the testbed for the Frontier exascale system, using 18 to 384 Graphics Compute Dies on AMD MI250X GPUs. We compare strong- and weak-scaling capabilities, linear solver performance, and time to solution. We also identify leading inhibitors to parallel scaling.

17 WIND ENERGY↗

High-Order Hybrid RANS-LES Study of NACA0012 Wing Sections

We develop hybrid RANS-LES strategies within Nek5000 for application to airfoil sections at small flight configurations. We present a validation and verification study of $k \space – \space \tau$ SST applied to a NACA 0012 wing section in a pure RANS and in a hybrid RANS-LES setup. The study shows good corroboration with existing experimental and numerical datasets. We also analyze some of the observed discrepancies with the experiments by evaluating the side wall “blocking” effect. We demonstrate that for the hybrid turbulence modeling approach a high-order spectral- element discretization converges faster (i.e., with less resolution) than a representative low-order finite-volume-based approach.

42 ENGINEERING↗

Energy Exascale Computational Fluid Dynamics Simulations With the Spectral Element Method

Development and application of the open-source GPU-based fluid-thermal simulation code, NekRS, are described. Time advancement is based on an efficient kth-order accurate timesplit formulation coupled with scalable iterative solvers. Spatial discretization is based on the high-order spectral element method (SEM), which affords the use of fast, low-memory, matrix-free operator evaluation. Further, recent developments include support for nonconforming meshes using overset grids and for GPU-based Lagrangian particle tracking. Results of large-eddy simulations of atmospheric boundary layers for wind-energy applications as well as extensive nuclear energy applications are presented.

42 ENGINEERING↗

Exascale Multiphysics Nuclear Reactor Simulations for Advanced Designs

ENRICO is a coupled application developed under the U.S. Department of Energy's Exascale Computing Project (ECP) targeting the modeling of advanced nuclear reactors. It couples radiation transport with heat and fluid simulation, including the high-fidelity, highresolution Monte-Carlo code Shift and the Computational fluid dynamics code NekRS. NekRS is a highly-performant open-source code for simulation of incompressible and low-Mach fluid flow, heat transfer, and combustion with a particular focus on turbulent flows in complex domains. It is based on rapidly convergent high-order spectral element discretizations that feature minimal numerical dissipation and dispersion. State-of-the-art multilevel preconditioners, efficient high-order time-splitting methods, and runtime-adaptive communication strategies are built on a fast OCCA-based kernel library, libParanumal, to provide scalability and portability across the spectrum of current and future high-performance computing platforms. On Frontier, Nek5000/RS has recently achieved an unprecedented milestone in breaching over 1 billion spectral elements and 350 billion degrees of freedom. Shift has demonstrated the capability to transport upwards of 1 billion particles per second in full core nuclear reactor simulations featuring complete temperature-dependent, continuous-energy physics on Frontier. Shift achieved a weak-scaling efficiency of 97.8% on 8192 nodes of Frontier and calculated 6 reactions in 214,896 fuel pin regions below 1% statistical error yielding first-of-a-kind resolution for a Monte Carlo transport application.

Hamilton, Steven P.↗

Development of MHD for NekRS

A spectral-element-based formulation of incompressible MHD is presented in the context of the open-source fluid-thermal code, Nek5000/RS. The formulation supports magnetic fields in a solid domain that surrounds the fluid domain. Several steady-state and time-transient model problems are presented as part of the code verification process. Nek5000/RS is designed for large-scale turbulence simulations, which will be the next step with this new MHD capability.

97 MATHEMATICS AND COMPUTING↗

Advanced Turbulence Models for Large-Scale Atmospheric Boundary Layer Flows

We present high-fidelity large-eddy-simulation (LES) modeling approaches for the turbulent atmospheric boundary layer (ABL) flows. Wind energy is a prime example of an application driven by ABL. Generation of electrical energy from farms of wind turbines at night in the stable ABL is a particularly interesting situation. In this report, we consider the well-known GEWEX (Global Energy and Water Cycle Experiment) Atmospheric Boundary Layer Study (GABLS) stably stratified benchmark LES case. We use a high-order spectral element code Nek5000/RS, which is supported under the DOE's Exascale Computing Project (ECP) Center for Efficient Exascale Discretizations (CEED) project, targeting application simulations on various acceleration-device based exascale computing platforms. In our earlier ANL report, we demonstrated our newly developed subgrid-scale (SGS) models based on high-pass filter (HPF), mean-field eddy viscosity (MFEV), and Smagorinsky (SMG) with no-slip and traction boundary conditions, provided with low-order statistics, convergence and turbulent structure analysis. In this report, we extend the range of our SGS modeling approaches in the context of the mean-field eddy viscosity (MFEV), to include the solution of an SGS turbulent kinetic energy equation (TKE). We demonstrate the model fidelity of Nek5000/RS in comparison to that of AMR-Wind, a block-structured second-order finite-volume code with adaptive-mesh-refinement capabilities, with which we studied scaling performance for both codes in comparison on DOE's leadership computing platforms.

17 WIND ENERGY↗

Turbulence Modeling with Nek5000/RS, SOD2D and Alya

We present Validation and Verification (V & V) study of two high-order spectral element method (SEM) based Computational Fluid Dyanimcs (CFD) codes that we will utilize for work on turbulence modeling: Nek5000 and SOD2D. While the former solves the incompressible form of Navier-Stokes equation, the latter works with compressible set of equations and uses an entropy-viscosity formulation to account for the discontinuities for high Mach number flows. We demonstrate the accuracy of these codes for two benchmark sub-sonic turbulent flows: periodic channel and pipe flow, by carrying out first and second order statistical analysis, grid convergence and turbulent structure analysis using wall-resolved large eddy simulations (WRLES). Later, we report on the implementation and testing of various wall-modeling strategies for large eddy simulation of turbulent flows in Nek5000. These include both classical log-law based and decision-tree based machine-learning models. Accuracy of these closure strategies are analyzed and necessary future work is outlined.

97 MATHEMATICS AND COMPUTING↗

NekRS, a GPU-accelerated spectral element Navier–Stokes solver

The development of NekRS, a GPU-oriented thermal-fluids simulation code based on the spectral element method (SEM) is described. For performance portability, the code is based on the open concurrent compute abstraction and leverages scalable developments in the SEM code Nek5000 and in libParanumal, which is a library of high-performance kernels for high-order discretizations and PDE-based miniapps. Critical performance sections of the Navier–Stokes time advancement are addressed. Performance results on several platforms are presented here, including scaling to 27,648 V100s on OLCF Summit, for calculations of up to 60B gridpoints.

97 MATHEMATICS AND COMPUTING↗

Simulating Atmospheric Boundary Layer Turbulence with Nek5000/RS

We present large-eddy-simulation (LES) modeling approaches for the simulation of atmospheric boundary layer turbulence that are of direct relevance to wind energy production. In this report, we study a GABLS benchmark problem using high-order spectral element code Nek5000/RS, which is supported under the DOE’s Exascale Computing Project (ECP) Center for Efficient Exascale Discretizations (CEED) project, targeting application simulations on various acceleration-device based exascale computing platforms [1, 2]. We demonstrate our newly developed subgrid-scale (SGS) models based on high-pass filter (HPF), mean-field eddy viscosity (MFEV), and Smagorinsky (SMG) with no-slip and traction boundary conditions, provided with low-order statistics, convergence and turbulent structure analysis. The model fidelity and scaling performance of Nek5000/RS on DOE’s leadership computing platforms in comparison to those of AMR-Wind, a block-structured second-order finite-volume code with adaptive-mesh-refinement capabilities, are discussed in [3].

17 WIND ENERGY↗

Nek5000/RS Performance on Advanced GPU Architectures

We demonstrate NekRS performance results on various advanced GPU architectures. NekRS is a GPU-accelerated version of Nek5000 that is targeting high performance on forthcoming exascale platforms. It is being developed in DOE’s Center of Efficient Exascale Discretizations (CEED), which is one of the co-design centers under the Exascale Computing Project (ECP). In this report, we consider Frontier, Crusher, Spock, Polaris, Perlmutter, ThetaGPU and Summit. Simulations are performed using ExaSMR’s 17x17 rod-bundle geometries with different problem sizes. The report focuses on strong scaling performance and analysis. Many of the results shown in this report are the outcome from participation in the ALCF GPU Hackathon on Polaris, which was held on 7/19/22, and 7/26–7/28/22. Members of the Nek5000/RS team included Misun Min, Yu-Hsiang Lan, Paul Fischer and Thilina Rathnayake. Mentors were Kris Rowe (ALCF) and Peng Wang (NVIDIA). Also presented in this report are results on Frontier, which were obtained in collaboration with John Holmen at OLCF.

97 MATHEMATICS AND COMPUTING↗

Pressure Drop Correlation Improvement for the Near-Wall Region of Pebble-Bed Reactors

Packed beds play an important role in several engineering fields, with their applications in nuclear energy being driven by the development of next-generation reactors utilizing pebble fuel. The random nature of a packed pebble bed creates a flow field that is complex and difficult to predict. Porous media models are an attractive option for modeling pebble-bed reactors (PBRs), as they provide intermediate fidelity results and are computationally efficient. Porous media models, however, rely on the use of correlations to estimate the effect of complicated flow features on the pressure drop and heat transfer in the system. Existing correlations were developed to predict the average behavior of the bed, but they are inaccurate in the near-wall region where the presence of the wall affects the pebble packing. This work aims to investigate the accuracy of a porous media model using the Kerntechnischer Ausschuss (KTA) correlation, the most common pressure drop correlation for PBRs compared to the high-fidelity large eddy simulation (LES). A bed of 1568 pebbles is investigated at Reynolds numbers from 625 to 10 000. The bed is divided into five concentric subdomains to compare the average velocity, friction losses, and form losses between the porous media and LES codes. The comparison between the LES simulation and the KTA correlation revealed that the KTA correlation largely underpredicts the form losses in the near-wall region, leading to an overprediction of the velocity near the wall by nearly 30%. An investigation of the form losses across the range of Reynolds numbers in the LES results provided additional insight into how the KTA correlation may be improved to better predict these spatial effects in a pebble bed. These data suggest that the form coefficient near the wall must be increased by 48% while decreasing the form coefficient of the inner bulk region of the bed by 15%. The implementation of these improvements to the KTA correlation in a porous media model produced a radial velocity profile that saw significantly improved agreement with the LES results.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Scalable Interpolation on GPUs for Thermal Fluids Applications

The need for efficient interpolation arises in many use cases for spectral element simulations. Towards this end, we present a fast and robust GPU-enable interpolation utility for NekRS, a spectral-element, Navier-Stokes solver. This utility is based on the findpts and findpts eval routines from the gslib library. To demonstrate the functionality, we added support for particle tracking. The GPU implementation of the interpolation allows for tracking 100 times more particles for the same computational cost.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Port and optimize the CEED software stack to Aurora/Frontier EA (ECP Milestone Report)

The goal of this milestone was to port the CEED software stack, including Nek, MFEM and libCEED to the Frontier and Aurora early access hardware, work on optimizing the performance on AMD and Intel GPUs, and demonstrate impact in CEED-enabled ECP applications. As part of this milestone, we also performed a number of other activities, including continued optimizations for CEED applications at scale on Summit, performance evaluation on non-ECP hardware, including the Fugaku’s A64FX chip and the NVIDIA A100 GPU architecture, exploring the use of just-in-time compilations in applications at scale and potential of mixed precision optimizations in the CEED discretization and solver algorithms, and more. During the milestone period, the CEED team released new versions of 6 of its packages, including major releases of MFEM, NekRS and OCCA. We also organized the fifth CEED Annual meeting (CEED5AM) which included nearly 100 researchers from national labs, universities and industry.

97 MATHEMATICS AND COMPUTING↗

GPU algorithms for Efficient Exascale Discretizations

In this paper we describe the research and development activities in the Center for Efficient Exascale Discretization within the US Exascale Computing Project, targeting state-of-the-art high-order finite-element algorithms for high-order applications on GPU-accelerated platforms. Furthermore, we discuss the GPU developments in several components of the CEED software stack, including the libCEED, MAGMA, MFEM, libParanumal, and Nek projects. We report performance and capability improvements in several CEED-enabled applications on both NVIDIA and AMD GPU systems.

97 MATHEMATICS AND COMPUTING↗

Efficient exascale discretizations: High-order finite element methods

Efficient exploitation of exascale architectures requires rethinking of the numerical algorithms used in many large-scale applications. These architectures favor algorithms that expose ultra fine-grain parallelism and maximize the ratio of floating point operations to energy intensive data movement. One of the few viable approaches to achieve high efficiency in the area of PDE discretizations on unstructured grids is to use matrix-free/partially assembled high-order finite element methods, since these methods can increase the accuracy and/or lower the computational time due to reduced data motion. In this paper we provide an overview of the research and development activities in the Center for Efficient Exascale Discretizations (CEED), a co-design center in the Exascale Computing Project that is focused on the development of next-generation discretization software and algorithms to enable a wide range of finite element applications to run efficiently on future hardware. CEED is a research partnership involving more than 30 computational scientists from two US national labs and five universities, including members of the Nek5000, MFEM, MAGMA and PETSc projects. We discuss the CEED co-design activities based on targeted benchmarks, miniapps and discretization libraries and our work on performance optimizations for large-scale GPU architectures. We also provide a broad overview of research and development activities in areas such as unstructured adaptive mesh refinement algorithms, matrix-free linear solvers, high-order data visualization, and list examples of collaborations with several ECP and external applications.

97 MATHEMATICS AND COMPUTING↗

Feasibility of full-core pin resolved CFD simulations of small modular reactor with momentum sources

Complex flow structure interactions and heat transfer processes take place in nuclear reactor cores. Given the extreme pressure/temperature and radioactive conditions inside the core, numerical simulations offer an attractive and sometimes more feasible approach to study the related flow and heat transfer phenomena in addition to the experiments. Under the Exascale Computing Project, the full-core simulation of a small modular reactor (SMR) has been pursued coupling Computational Fluid Dynamics (CFD) and neutronics. A key aspect of the modeling of SMR fuel assemblies is the presence of spacer grids and the mixing promoted by mixing vanes or the equivalent. A reduced order methodology is adopted based on momentum sources to mimic the mixing of the vanes. The momentum sources have been carefully calibrated with detailed Large Eddy Simulations (LES) of spacer grids performed with Nek5000. Modeling the spacer grid and mixing vanes (SGMV) effect without body-fitted computational grid avoids the excessive costs in resolving the local geometric details, and thus supports the simulation to be scaled up to the full core. Besides the progress on momentum source modeling, this paper also features the first full-core pin resolved CFD simulation ever performed to the authors' knowledge. This represents a significant advancement in capability for the CFD of nuclear reactors, which will hopefully serve as an inspiration for further integrating high-fidelity numerical simulations in actual engineering designs.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗