Engineering Papers⌕ Search

Engineering topics

Tomboulides, Ananias

Publications and source records attributed to Tomboulides, Ananias.

At least 19 records

Towards exascale for wind energy simulations

We examine large-eddy-simulation modeling approaches and computational performance of two open-source computational fluid dynamics codes for the simulation of atmospheric boundary layer flows that are of direct relevance to wind energy production. The first code, NekRS, is a high-order, unstructured-grid, spectral element code. The second code, AMR-Wind, is a second-order, block-structured, finite-volume code with adaptive mesh refinement capabilities. The objective of this study is to co-develop these codes in order to improve model fidelity and performance for each. These features will be critical for running ABL-based applications such as wind farm analysis on advanced computing architectures. To this end, we investigate the performance of NekRS and AMR-Wind on the Oak Ridge Leadership Facility supercomputers Summit, using 4 to 800 nodes (24 to 4,800 NVIDIA V100 GPUs), and Crusher, the testbed for the Frontier exascale system, using 18 to 384 Graphics Compute Dies on AMD MI250X GPUs. We compare strong- and weak-scaling capabilities, linear solver performance, and time to solution. We also identify leading inhibitors to parallel scaling.

17 WIND ENERGY↗

CFD simulations of Molten Salt Fast Reactor core cavity flows

Computational Fluid Dynamics (CFD) has become increasingly important in the research and development of advanced nuclear reactors. Here, in the current study, extensive CFD simulations were conducted for the coolant flow in Molten Salt Fast Reactor (MSFR) core models using the state-of-the-art spectral element flow solver Nek5000 and multiscale coarse-mesh thermal-hydraulic software Pronghorn. The underlying motivation is to seek an in-depth understanding of how the internal velocity distribution can be influenced by the MSFR core cavity shape, the Reynolds number, turbulence modeling options and the inlet boundary conditions. The CFD techniques involved in this investigation range from coarse-mesh CFD, RANS modeling, to the high-fidelity LES calculations. Specifically, a series of RANS simulations were performed for the 2-D axisymmetric core model and 3-D wedge domains to study the flow distribution inside the MSFR core. It is observed that a proper representation of the MSFR inlet channel duct is important for the prediction of internal flow distribution. It is also showcased here how researchers can leverage the Nek5000 CFD results to calibrate more efficient coarse-mesh CFD tools, like Pronghorn, for the actual MSFR design needs. Moreover, this paper highlights a 3-D LES model for an entire MSFR core using the spectral element method and demonstrates the feasibility of this modeling approach. The readiness and potential limitations of the RANS approach are examined with respect to the high-fidelity LES simulations. The present investigation lays a solid foundation as we are leveraging the high-fidelity CFD capabilities to inform MSFR design efforts.

97 MATHEMATICS AND COMPUTING↗

Verification and Validation of Spectral Element Code for Supercritical CO2 Flow in Vertical Heated Tubes

The investigation of heat transfer in supercritical CO2 (sCO2) has garnered considerable attention in recent decades, given sCO2's potential as a promising working fluid for advanced power conversion cycles. Despite previous research efforts, there are still gaps in our understanding of sCO2 heat transfer, particularly in conditions associated with heat transfer deterioration. To delve into sCO2 heat transfer more comprehensively, we propose employing the high-fidelity computational fluid dynamics code NekRS to simulate sCO2 flow using the large eddy simulation technique. Through graphics processing unit acceleration, NekRS achieves a higher computational speed than traditional CPU-based systems. However, before using NekRS in practical applications involving sCO2, it is imperative to perform verification and validation. Here, this paper presents our efforts to verify and validate the NekRS code's capability for simulating sCO2 using heated vertical tubes, where heat transfer deterioration usually happens. To accommodate the unique properties of sCO2, we have modified the NekRS code by integrating third-party property modules, such as REFPROP and PROPATH. Our simulations are compared with experimental and numerical data from the literature, instilling confidence in leveraging NekRS for future engineering applications. Our simulations also reveal that the accuracy of the property module significantly impacts the results, with REFPROP outperforming PROPATH for sCO2 properties. Additionally, we observed that, depending on the flow direction, buoyancy can either enhance or suppress turbulence in sCO2 flow. In upward flow, under certain conditions, the suppressed turbulence leads to heat transfer deterioration, resulting in elevated wall temperatures.

NekRS↗

Energy Exascale Computational Fluid Dynamics Simulations With the Spectral Element Method

Development and application of the open-source GPU-based fluid-thermal simulation code, NekRS, are described. Time advancement is based on an efficient kth-order accurate timesplit formulation coupled with scalable iterative solvers. Spatial discretization is based on the high-order spectral element method (SEM), which affords the use of fast, low-memory, matrix-free operator evaluation. Further, recent developments include support for nonconforming meshes using overset grids and for GPU-based Lagrangian particle tracking. Results of large-eddy simulations of atmospheric boundary layers for wind-energy applications as well as extensive nuclear energy applications are presented.

42 ENGINEERING↗

Turbulence Modeling with Nek5000/RS, SOD2D and Alya

We present Validation and Verification (V & V) study of two high-order spectral element method (SEM) based Computational Fluid Dyanimcs (CFD) codes that we will utilize for work on turbulence modeling: Nek5000 and SOD2D. While the former solves the incompressible form of Navier-Stokes equation, the latter works with compressible set of equations and uses an entropy-viscosity formulation to account for the discontinuities for high Mach number flows. We demonstrate the accuracy of these codes for two benchmark sub-sonic turbulent flows: periodic channel and pipe flow, by carrying out first and second order statistical analysis, grid convergence and turbulent structure analysis using wall-resolved large eddy simulations (WRLES). Later, we report on the implementation and testing of various wall-modeling strategies for large eddy simulation of turbulent flows in Nek5000. These include both classical log-law based and decision-tree based machine-learning models. Accuracy of these closure strategies are analyzed and necessary future work is outlined.

97 MATHEMATICS AND COMPUTING↗

Stakeholder-Driven Improvements to Nek5000 and NekRS

As a key part of the U.S. Department of Energy’s Nuclear Energy Advanced Modeling and Simulation program (NEAMS), the next generation of tools for nuclear reactor design are being developed. A primary metric for success of this program is the adoption of those tools by industry stakeholders. Based on feedback provided directly to NEAMS, two of the major weaknesses have been a lack of moderate-fidelity Reynolds-averaged Navier-Stokes (RANS) turbulence models and available documentation. Over the past several years, developments have been focused on addressing the needs of those stakeholders. The outcome of those efforts are described herein.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Demonstration of RANS models with wall functions in the spectral element code Nek5000

The spectral element based computational fluid dynamics (CFD) code Nek5000 has been traditionally used for high-fidelity applications, such as direct numerical simulation (DNS) and large eddy simulation (LES). These techniques require very fine numerical resolution to accurately capture turbulent fluctuations which can be prohibitively expensive for users without access to leadership class computing facilities. For broader application and adoption, significant effort has been invested to develop Reynolds-averaged Navier–Stokes (RANS) capabilities in Nek5000. Here, this work presents details of the implementation and demonstration of the standard wall functions for the κ–τ model in Nek5000. Results using the wall-modeled approach are compared to a wall-resolved approach for cases with negligible pressure gradient, viz., channel flow, pipe flow and flow in a reactor subchannel. Results show reasonably good agreement between the two approaches for friction factor and Nusselt number. Some expected differences are identified near the wall. These cases demonstrate the potential for significant computational savings by using much coarser meshes for the wall-modeled approach, with only minor differences between the predicted result. Additionally, several Reynolds numbers up to 1,000,000 are demonstrated for pipe flow and predicted friction factors and Nusselt numbers compared well to available correlations, with the worst below 10%. As the Reynolds number is increased, better agreement is observed between the correlations and the wall-modeled approach. In addition, flow in a molten salt fast reactor (MSFR) core is considered which features an adverse pressure gradient and flow separation. It showcases the inability of standard wall functions to accurately predict flows with adverse pressure gradients. The results, however, match reasonably well in trend in regions of the flow where the boundary layer is attached. Ongoing research is dedicated to include a pressure gradient correction to wall functions to improve the accuracy of flows with separation or reattachment and adverse or favorable pressure gradients.

42 ENGINEERING↗

NekRS, a GPU-accelerated spectral element Navier–Stokes solver

The development of NekRS, a GPU-oriented thermal-fluids simulation code based on the spectral element method (SEM) is described. For performance portability, the code is based on the open concurrent compute abstraction and leverages scalable developments in the SEM code Nek5000 and in libParanumal, which is a library of high-performance kernels for high-order discretizations and PDE-based miniapps. Critical performance sections of the Navier–Stokes time advancement are addressed. Performance results on several platforms are presented here, including scaling to 27,648 V100s on OLCF Summit, for calculations of up to 60B gridpoints.

97 MATHEMATICS AND COMPUTING↗

Updates to Nek5000: RANS wall functions, NRC support, and documentation

As a key part of the U.S. Department of Energy’s Nuclear Energy Advanced Modeling and Simulation program (NEAMS), the next generation of tools for nuclear reactor design are being developed. A primary metric for success of this program is the adoption of those tools by industry stakeholders. To help meet this metric for Nek5000, development efforts this year have focused on the expansion of the available Reynolds-averaged Navier-Stokes (RANS) turbulence models to include wall function models, continued support of the U.S. Nuclear Regulatory Commission in simulating a hydrogen mitigation benchmark experiment, and expansion of the Nek5000 documentation to enhance the code’s usability. The implementation of wall functions is intended to significantly decrease time to-solution for a wide range of problems and it has been identified as an important feature by stakeholders across the nuclear community. The initial implementation of standard and pressure corrected wall functions has been tested in Nek5000 on the classical problems of channel flow and a backward facing step. It has also been tested on a 2D molten salt fast reactor core and a T-junction. The recommended formulation of choosing y + = 30 as a boundary value combined with a Neumann-Neumann formulation of the boundary condition for k and τ produces the most consistent results compared to a wall resolved approach. The collaboration with the NRC is a continuing exercise that has been ongoing for multiple years. This year we have concluded investigations into appropriate inlet conditions, showing that fully developed turbulent conditions are adequate for the full PANDA domain. We have also concluded simulations for an unobstructed jet and are now refocusing to the original obstructed jet case. The expansion of the code documentation has also been an ongoing effort over the past few years to address feedback from the various stakeholders. The documentation now more fully covers many of the available features and closes identified SQA gaps. Finally we report on three training sessions that were offered over the past year in an effort to expand the Nek5000 user base.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

CFD Simulations to Support Pronghorn Modeling of a Molten Salt Fast Reactor

To assist with the deployment of next generation molten salt reactors (MSRs), the DOE NEAMS program is developing advanced simulation capabilities. As part of this work, the high-fidelity CFD tool, Nek5000 was used to inform model development for the engineering scale tool, Pronghorn. The particular case chosen was the EVOL molten salt fast reactor concept. This concept uses an open core design, which is well known to be sensitive to the particular geometry and can have complex flow behavior that varies across a range of conditions. This makes it an idea candidate for demonstration of the Hi2Lo concept of using higher fidelity models to benchmark lower fidelity models. Simulations of the EVOL design were performed using both LES (high-fidelity) and RANS (moderate-fidelity) in Nek5000. The LES results indicated discrepancies in 2-D axisymmetric RANS, indicating that the core is better modeled in RANS with a 3-D wedge representation of a portion of the core, which can accurately account for the effect of the inlet channels. Finally, results from the 3-D RANS were used to modify an existing turbulence model in Pronghorn. Improvement in the calibrated Pronghorn model was demonstrated. Future work is suggested to focus on expanding the LES calculation to provide a more direct point of comparison for both the RANS and Pronghorn models as well as incorporating more of the relevant MSR physics (such as delayed neutron precursor tracking) into the model.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

Nek5000: improvements in the available RANS models, meshing, tutorials, and training

This year, the Nuclear Energy Advanced Modeling Simulation program (NEAMS) thermal-hydraulics report for Nek5000 NRC- and verification and validation (V&V)-driven development focuses on following areas of code application and improvement. First we have continued improvements of RANS modeling capabilities in Nek5000 including improved k-tau model focusing mostly on wallfunction initial implementation with spectral element method (SEM) and initiating investigation of an alternative approach XSEM that greatly reduces discretization errors.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

GPU algorithms for Efficient Exascale Discretizations

In this paper we describe the research and development activities in the Center for Efficient Exascale Discretization within the US Exascale Computing Project, targeting state-of-the-art high-order finite-element algorithms for high-order applications on GPU-accelerated platforms. Furthermore, we discuss the GPU developments in several components of the CEED software stack, including the libCEED, MAGMA, MFEM, libParanumal, and Nek projects. We report performance and capability improvements in several CEED-enabled applications on both NVIDIA and AMD GPU systems.

97 MATHEMATICS AND COMPUTING↗

Efficient exascale discretizations: High-order finite element methods

Efficient exploitation of exascale architectures requires rethinking of the numerical algorithms used in many large-scale applications. These architectures favor algorithms that expose ultra fine-grain parallelism and maximize the ratio of floating point operations to energy intensive data movement. One of the few viable approaches to achieve high efficiency in the area of PDE discretizations on unstructured grids is to use matrix-free/partially assembled high-order finite element methods, since these methods can increase the accuracy and/or lower the computational time due to reduced data motion. In this paper we provide an overview of the research and development activities in the Center for Efficient Exascale Discretizations (CEED), a co-design center in the Exascale Computing Project that is focused on the development of next-generation discretization software and algorithms to enable a wide range of finite element applications to run efficiently on future hardware. CEED is a research partnership involving more than 30 computational scientists from two US national labs and five universities, including members of the Nek5000, MFEM, MAGMA and PETSc projects. We discuss the CEED co-design activities based on targeted benchmarks, miniapps and discretization libraries and our work on performance optimizations for large-scale GPU architectures. We also provide a broad overview of research and development activities in areas such as unstructured adaptive mesh refinement algorithms, matrix-free linear solvers, high-order data visualization, and list examples of collaborations with several ECP and external applications.

97 MATHEMATICS AND COMPUTING↗

Feasibility of full-core pin resolved CFD simulations of small modular reactor with momentum sources

Complex flow structure interactions and heat transfer processes take place in nuclear reactor cores. Given the extreme pressure/temperature and radioactive conditions inside the core, numerical simulations offer an attractive and sometimes more feasible approach to study the related flow and heat transfer phenomena in addition to the experiments. Under the Exascale Computing Project, the full-core simulation of a small modular reactor (SMR) has been pursued coupling Computational Fluid Dynamics (CFD) and neutronics. A key aspect of the modeling of SMR fuel assemblies is the presence of spacer grids and the mixing promoted by mixing vanes or the equivalent. A reduced order methodology is adopted based on momentum sources to mimic the mixing of the vanes. The momentum sources have been carefully calibrated with detailed Large Eddy Simulations (LES) of spacer grids performed with Nek5000. Modeling the spacer grid and mixing vanes (SGMV) effect without body-fitted computational grid avoids the excessive costs in resolving the local geometric details, and thus supports the simulation to be scaled up to the full core. Besides the progress on momentum source modeling, this paper also features the first full-core pin resolved CFD simulation ever performed to the authors' knowledge. This represents a significant advancement in capability for the CFD of nuclear reactors, which will hopefully serve as an inspiration for further integrating high-fidelity numerical simulations in actual engineering designs.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Nek5000 developments in support of industry and the NRC

This year, the Nuclear Energy Advanced Modeling Simulation program (NEAMS) thermal-hydraulics verification and validation (V&V) work has focused in three areas of Nek5000 V&V-driven development. First, in a close collaborative effort with the U. S. Nuclear Regulatory Commission (NRC) staff, we have continued V&V efforts for the HYMERES-2 project using the OECD/NEA sponsored testing in the PSI PANDA facility. This year’s focus of ANL-NRC collaboration involves Nek5000 setups and validation for a range of problems relevant to and including the HYMERES-2 benchmark from PSI. The primary outcome of this year efforts is a more efficient geometry and inlet modeling simplification after a careful sensitivity study of the inlet profiles and pipe geometries. The resulting modeling choice of a short recycling/fully-developed turbulent inlet is within the experimental uncertainty estimate. This finding simplifies the next step of the cross-V&V HYMERES-2 project. In addition, the ANL team continue to provide assistance to the NRC staff in the form of Nek5000 application support in general and on the use of the HPC platforms of ALCF and INL in particular. This supports the NRC’s assessment of Nek5000 for use with the NRC Blue CRAB code suite. Second, we have implemented and tested more robust model of URANS, namely the k – τ model, a variant of the k-ω model, along with other improvements to RANS Nek5000 modeling in general. Because of its demonstrated robustness and stability, the k – τ model is the only RANS model that has been implemented in the new GPU version of the Nek5000 code, nekRS. Lastly, we report the initial implementation of Jacobian-free Newton Krylov approach to the direct Newton method for steady fluid solvers aimed at acceleration of RANS modeling and at IC improvement for LES campaigns. Also leveraging the Exascale Computing Project (ECP) ANL/CEED & SMR team’s software development effort to support NEAMS problems at large scale of the advanced computing architectures, NekRS, a GPU variant of Nek5000, built on top of kernels from libParanumal using OCCA for portability, has been successfully run on the full system of Summit (4608 nodes, 27648 GPUs).

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Support CEED-enabled ECP applications in their preparation for Aurora/Frontier

The goal of this milestone was to help CEED-enabled ECP applications (particularly ExaSMR, MARBL, ExaAM, ExaWind and E3SM) in their preparations for the Aurora and Frontier architectures. This work included collaboration with ECP vendors and porting and optimization of CEED’s benchmarks and miniapps to early access hardware. As part of this milestone, we also made the best bake-off problems and bake-off kernel implementation from Nek, MFEM, libParanumal and the external community available in the latest libCEED release, libCEED-0.7. During the milestone period we also organized, in virtual form, the fourth CEED Annual meeting (CEED4AM) which included representatives from ECP applications, vendors and software technology projects. The specific tasks addressed in this milestone were to: (1) Work with vendors to port and run CEED benchmarks on early access systems for Aurora and Frontier; (2) Make the best BP/BK implementations from Nek, MFEM, libParanumal and external community available in libCEED; (3) Organize the next CEED Annual meeting (CEED4AM); and (4) Optimize CEED applications and miniapps for Aurora and Frontier architectures. The artifacts delivered include the next libCEED release, libCEED-0.7, and a number of developments integrated within applications to improve their GPU performance and capabilities. See the CEED website, http://ceed.exascaleproject.org and the CEED GitHub organization, http://github.com/ceed for more details.

97 MATHEMATICS AND COMPUTING↗

CEED-MS34: Improve performance and capabilities of CEED-enabled ECP applications on Summit/Sierra. Exascale Computing Project Milestone Report

The main goal of this milestone was to help CEED-enabled ECP applications, including ExaSMR, MARBL, ExaWind and ExaAM, to improve their performance and capabilities on GPU systems like Summit and Lassen/Sierra. In addition, the CEED team also worked to: add and improve support for additional hardware and programming models in the CEED software components; release the next version of the CEED software stack, CEED-3.0; and demonstrate performance of libParanumal kernels in libCEED, Nek and MFEM. These additional tasks contributed directly to the main CEED-MS34 goal and will also play an important role in CEED’s future milestones.

97 MATHEMATICS AND COMPUTING↗