Engineering Papers⌕ Search

Engineering topics

Min, Misun

Publications and source records attributed to Min, Misun.

At least 37 records · Page 2

High-order algorithmic developments and optimizations for large-scale GPU-accelerated simulations (Milestone CEED-MS36)

The goal of this milestone was to improve the high-order software ecosystem for CEED-enabled ECP applications by making progress on efficient matrix-free kernels targeting forthcoming ECP architectures. These kernels included matrix-free preconditioning and the development of new set of CEED solver bake-off problems. As part of this milestone, we also released the next version of the CEED software stack, CEED-4.0, reported on results from several application collaborations, and documented the efforts of porting to AMD GPUs for Frontier and other modern architectures, such as Fugaku. The specific tasks addressed in this milestone were: (1) Port and run CEED benchmarks/miniapps on Frontier EA systems; (2) Demonstrate performant libCEED integration in MFEM, Nek and applications; (3) Matrix-free preconditioning of high-order operators; (4) Benchmark problems for fast high-order solvers on GPU platforms; and (5) Public release of CEED-4.0. The artifacts delivered include the next version of the CEED software stack, CEED-4.0, the next libCEED release, libCEED-0.8, and a number of developments integrated within applications to improve their GPU and CPU performance and capabilities. See the CEED website, https://ceed.exascaleproject.org and the CEED GitHub organization, https://github.com/ceed for more details.

97 MATHEMATICS AND COMPUTING↗

Cardinal: A Lower-Length-Scale Multiphysics Simulator for Pebble-Bed Reactors

This paper demonstrates a multiphysics solver for pebble-bed reactors, in particular, for Berkeley’s pebble-bed -fluoride-salt-cooled high-temperature reactor (PB-FHR) (Mark I design). The FHR is a class of advanced nuclear reactors that combines the robust coated particle fuel form from high-temperature gas-cooled reactors, the direct reactor auxiliary cooling system passive decay removal of liquid-metal fast reactors, and the transparent, high-volumetric heat capacitance liquid-fluoride salt working fluids (e.g., FLiBe) from molten salt reactors. This fuel and coolant combination enables FHRs to operate in a high-temperature, low-pressure design space that has beneficial safety and economic implications. The PB-FHR relies on a pebble-bed approach, and pebble-bed reactors are, in a sense, the poster child for multiscale analysis. Relying heavily on the MultiApp capability of the Multiphysics Object-Oriented Simulation Environment (MOOSE), we have developed Cardinal, a new platform for lower-length-scale simulation of pebble-bed cores. The lower-length-scale simulator comprises three physics: neutronics (OpenMC), thermal fluids (Nek5000/NekRS), and fuel performance (BISON). Cardinal tightly couples all three physics and leverages advances in MOOSE, such as the MultiApp system and the concept of MOOSE-wrapped applications. Moreover, Cardinal can utilize graphics processing units for accelerating solutions. In this paper, we discuss the development of Cardinal and the verification and validation and demonstration simulations.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗

Initial full core SMR simulations with NekRS

This document describes the completion of a recent milestone of the ExaSMR program concerning "Full core simulations". The ExaSMR project is developing tools for coupling Monte Carlo (MC) radiation transport solvers to a computational fluid dynamics (CFD) solver. Work is based on the Shift MC, OpenMC MC, and Nek5000/NekRS CFD codes. As part of this milestone, a novel set of pin-resolved CFD full-core simulations have been performed for the first time. These simulations represent a significant increase in capability in what is now possible with CFD on pre-Exascale systems. The simulations have been performed with the spectral element solvers NekRS on the supercomputer Summit. Multiple simulations campaigns have been conducted: (1) LES simulations in bare bundle, (2) RANS simulations in bare bundles, (3) RANS simulations with momentum sources and (4) Conjugate heat transfer calculations.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗

Support CEED-enabled ECP applications in their preparation for Aurora/Frontier

The goal of this milestone was to help CEED-enabled ECP applications (particularly ExaSMR, MARBL, ExaAM, ExaWind and E3SM) in their preparations for the Aurora and Frontier architectures. This work included collaboration with ECP vendors and porting and optimization of CEED’s benchmarks and miniapps to early access hardware. As part of this milestone, we also made the best bake-off problems and bake-off kernel implementation from Nek, MFEM, libParanumal and the external community available in the latest libCEED release, libCEED-0.7. During the milestone period we also organized, in virtual form, the fourth CEED Annual meeting (CEED4AM) which included representatives from ECP applications, vendors and software technology projects. The specific tasks addressed in this milestone were to: (1) Work with vendors to port and run CEED benchmarks on early access systems for Aurora and Frontier; (2) Make the best BP/BK implementations from Nek, MFEM, libParanumal and external community available in libCEED; (3) Organize the next CEED Annual meeting (CEED4AM); and (4) Optimize CEED applications and miniapps for Aurora and Frontier architectures. The artifacts delivered include the next libCEED release, libCEED-0.7, and a number of developments integrated within applications to improve their GPU performance and capabilities. See the CEED website, http://ceed.exascaleproject.org and the CEED GitHub organization, http://github.com/ceed for more details.

97 MATHEMATICS AND COMPUTING↗

Toward Exascale: Overview of Large Eddy Simulations and Direct Numerical Simulations of Nuclear Reactor Flows with the Spectral Element Method in Nek5000

At the beginning of the last decade, Petascale supercomputers (i.e., computers capable of more than 1 petaFLOP) emerged. Now, at the dawn of exascale supercomputing, we provide a review of recent landmark simulations of portions of reactor components with turbulence-resolving techniques that this computational power has made possible. In fact, these simulations have provided invaluable insight into flow dynamics, which is difficult or often impossible to obtain with experiments alone. We focus on simulations performed with the spectral element method, as this method has emerged as a powerful tool to deliver massively parallel calculations at high fidelity by using large eddy simulation or direct numerical simulation. We also limit this paper to constant-property incompressible flow of a Newtonian fluid in the absence of other body or external forces, although the method is by no means limited to this class of flows. We briefly review the fundamentals of the method and the reasons it is compelling for the simulation of nuclear engineering flows. We review in detail a series of Petascale simulations, including the simulations of helical coil steam generators, fuel assemblies, and pebble beds. Even with Petascale computing, however, limitations for nuclear modeling and simulation tools remain. In particular, the size and scope of turbulence-resolving simulations are still limited by computing power and resolution requirements, which scale with the Reynolds number. In the final part of this paper, we discuss the future of the field, including recent advancements in emerging architectures such as GPUbased supercomputers, which are expected to power the next generation of high-performance computers.

computational fluid dynamics↗

Scalability of high-performance PDE solvers

Performance tests and analyses are critical to effective high-performance computing software development and are central components in the design and implementation of computational algorithms for achieving faster simulations on existing and future computing architectures for large-scale application problems. In this article, we explore performance and space-time trade-offs for important compute-intensive kernels of large-scale numerical solvers for partial differential equations (PDEs) that govern a wide range of physical applications. We consider a sequence of PDE-motivated bake-off problems designed to establish best practices for efficient high-order simulations across a variety of codes and platforms. We measure peak performance (degrees of freedom per second) on a fixed number of nodes and identify effective code optimization strategies for each architecture. In addition to peak performance, we identify the minimum time to solution at 80% parallel efficiency. The performance analysis is based on spectral and p-type finite elements but is equally applicable to a broad spectrum of numerical PDE discretizations, including finite difference, finite volume, and h-type finite elements.

97 MATHEMATICS AND COMPUTING↗

An Update on Cardinal: Toward Full Core Pebble Simulations

The new DOE-NE Center of Excellence for Thermal-fluids applications in Nuclear Energy inaugurated in April 2018 considers and researches novel new solution strategies for historically challenging flow issues that still plague the current fleet of deployed Light Water nuclear Reactors (LWRs) as well as predicting various fluid flow and fluid related issues with advanced reactor technologies. Our advanced thermal-fluids research and development approach synergistically combines three natural, though overlapping, length and time scales in a hierarchal multi-scale approach to avoid the temptation and pitfalls of attempting to develop a single solve-all algorithm for physical fluid flow problems that will span nine orders of magnitude in spatial and temporal scales. To demonstrate the multi-scale philosophy of the center we focus on Fluoride Cooled High Temperature Reactors (FHRs), and in particular on the Berkley’s PB-FHR Mark-I design. The Fluoride salt cooled High temperature Reactor (FHR) is a class of advanced nuclear reactors that combine the robust coated particle fuel form from high temperature gas cooled reactors, direct reactor auxiliary cooling system (DRACS) passive decay removal of liquid metal fast reactors, and the transparent, high volumetric heat capacitance liquid Fluoride salt working fluids - such as Flibe - from molten salt reactors. This combination of fuel and coolant enables FHRs to operate in a high-temperature low-pressure design space that has beneficial safety and economic implications. The PB-FHR reactor relies on a pebble bed approach and pebble bed reactors are in a sense the poster child for multi-scale analysis. The lower length-scale simulator for pebble reactor cores comprises three physics: neutronics (OpenMC), thermal-fluids (Nek5000/NekRS) and fuel performance (BISON). As part of Center of Excellence ongoing research efforts we have developed Cardinal, a new tool platform for lower length-scale simulation. Cardinal tightly couples all three physics and leverages advances in MOOSE such as the MultiApp system and the concept of MOOSE-wrapped Apps. The present report provides an update on the development of Cardinal with an extension of Cardinal to GPUs. We also perform a first-of-a-kind demonstration simulation on Summit representing a 10x capability increase in terms of pebble count for Cardinal.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗

CEED-MS34: Improve performance and capabilities of CEED-enabled ECP applications on Summit/Sierra. Exascale Computing Project Milestone Report

The main goal of this milestone was to help CEED-enabled ECP applications, including ExaSMR, MARBL, ExaWind and ExaAM, to improve their performance and capabilities on GPU systems like Summit and Lassen/Sierra. In addition, the CEED team also worked to: add and improve support for additional hardware and programming models in the CEED software components; release the next version of the CEED software stack, CEED-3.0; and demonstrate performance of libParanumal kernels in libCEED, Nek and MFEM. These additional tasks contributed directly to the main CEED-MS34 goal and will also play an important role in CEED’s future milestones.

97 MATHEMATICS AND COMPUTING↗

Performance tuning of CEED software and 1st and 2nd wave apps

The goal of this milestone was the performance tuning of the CEED software, as well as the use and tuning of CEED to accelerate the first and second wave of targeted ECP applications. In this milestone, the CEED team developed optimization techniques and tuned for performance the CEED software to accelerate the first and second wave target ECP applications. Specifically, the focus was on the following: (1) Efficient use of the memory sub-system for managing data motion and interactions among different physics packages and libraries; this included integration of the new developments in a libCEED 0.5 software release; (2) Enhancing the CEED libraries with GPU and AMD GPU support in close interaction with vendors; (3) Optimal data locality and motion, and enhanced scalability and parallelism; this included vendors interactions for improvements in data motion and making strong scaling easier and more efficient; and (4) Continue boosting performance for the first-wave and second-wave of ECP target applications, including ExaSMR, ExaWind, Urban, MARBL, E3SM, and ExaAM. A main part of this milestone was the performance tuning of the CEED first-wave and second-wave ECP applications. This included the ExaSMR application – Coupled Monte Carlo Neutronics and Fluid Flow Simulation of Small Modular Reactors (ORNL), the MARBL application – Next-Gen Multi-physics Simulation Code (LLNL), and the ExaAM ExaConstit – a miniapp for the Transforming Additive Manufacturing through Exascale Simulation applications project (ExaAM), as well as ExaWind, Urban, and E3SM. The artifacts delivered include performance improvements in CEED’s 1st and 2nd wave of applications, and tuned CEED software for various architectures through a number of backends, freely available in the CEED’s repository on GitHub. See the CEED website, http://ceed.exascaleproject.org and the CEED GitHub organization, http://github.com/ceed for more details. In addition to details and results from the above R&D efforts, in this document we are also reporting on other project-wide activities performed in Q4 of FY19 including: CEED’s third annual meeting, MFEM GPU kernels scaling improvements, libCEED 0.5 and NekRS releases, an overview paper covering latest results on a set of bake-off high-order finite element problems (BPs), and other outreach efforts.

97 MATHEMATICS AND COMPUTING↗

Nek5000 Enhancements for Faster Running Analysis

The goal of this milestone is to enhance the performance of Nek5000 for thermal-hydraulics analysis. In this milestone, the focus is to accelerate the time-to-solution for coupled calculations of turbulent flows and thermal variations having different time scales that which require resolution of fine-scale structures. The main artifacts delivered include the following: Steady state solvers for rod bundles and pebble beds; Enhancing RANS solvers with improved stability and performance. This document covers work primarily funded under the NEAMS program. We note that the achievements listed here would have been impossible without a strong collaboration between the NEAMS team and ECPCEED team.This synergy was enabled by algorithmic developments funded by DOE ECP Co-Design CEED and DOE ASCR Applied Math program. In this document we are reporting some details on the algorithmic strategies and their results.

47 OTHER INSTRUMENTATION↗

On the use of LES-based turbulent thermal-stress models for rod bundle simulations

An alternative methodology is proposed here to overcome the excessive cost of large eddy simulations (LES) of full-length heated rod bundle calculations, while improving the inaccurate results typically obtained with Reynolds-averaged Navier-Stokes equations (RANS). While the cost of the full-length LES is generally too high, LES of a small section of a single rod is usually affordable. The idea presented here consists of using the information granted by the small LES calculation to determine the appropriate turbulence viscosity or turbulent thermal diffusivity that can be used to solve only for the temperature field in a pseudo-RANS approach. The study has been performed with single-rod simulations with a P/D of 1.12 and 1.24, considering rod lengths that are representative of reactor applications, for the cases of uniform heat flux and a more realistic cosine-like axial heat distribution. Here, the spectral element code Nek5000 has been used for all LES, RANS, and pseudo-RANS simulations. The recently proposed Nek5000 steady-state solver has been used for solving the temperature field in the pseudo-RANS approach and has proved significantly faster than transient schemes. Prediction of thermal quantities is compared with classical linear and nonlinear RANS models. LES for the full-length rods has also been performed and is used as a reference. Results of the proposed method show significant improvements with respect to those obtained with RANS.

42 ENGINEERING↗

OpenACC acceleration for the P N – P N - 2 algorithm in Nek5000

Due to its high performance and throughput capabilities, GPU-accelerated computing is becoming a popular technology in scientific computing, in particular using programming models such as CUDA and OpenACC. The main advantage with OpenACC is that it enables to simply port codes in their ‘‘original’’ form to GPU systems through compiler directives, thus allowing an incremental approach. An OpenACC implementation is applied to the CFD code Nek5000 for simulation of incompressible flows, based on the spectral-element method. The work follows up previous implementations and focuses now on the P N - P N-2 method for the spatial discretization of the Navier–Stokes equations. Performance results of the ported code show a speed-up of up to 3.1 on multi-GPU for a polynomial order N > 11.

97 MATHEMATICS AND COMPUTING↗

Periodic corrugations to increase efficiency of thermophotovoltaic emitting structures

In this letter we consider the question of designing insulator/metal thermovoltaic structures with periodically corrugated interfaces that give optimal performance based on the metric of useful power density. Using a Monte Carlo approach in a robust, rapid, and high-accuracy numerical simulation strategy, we have identified such interface shapes. Here, we searched among the class of sinusoids and found that a flat-interface configuration could be significantly improved in transverse magnetic polarization. More specifically, we found that (i.) the performance improves with increasing corrugation amplitude (ii.) up to a maximum, (iii.) the shape of the corrugation is largely irrelevant, and (iv.) the period of the corrugation should be chosen in connection to the bandgap energy of the photovoltaic cell. For the latter we provide a simple expression as a starting point for practitioners interested in fabricating such structures.

36 MATERIALS SCIENCE↗

Engage second wave ECP/CEED applications

The CEED co-design center aims to impact a wide range of ECP application teams through focused 1-to-1 and 1-to many interactions, facilitated by CEED application liaisons, ultimately targeted to provide easy-to-use discretization libraries for high-order FEM-based application simulations. The goal of this milestone was to extend our reach by identifying a second wave of ECP applications that can benefit from the efficient discretization algorithms and software developed in CEED. These applications were added to the list of the CEED first wave ECP applications (ExaSMR and MARBL), which continue to be main targets for our work. As part of the milestone, we also appointed liaisons on the CEED team that will be responsible for engaging each application, identifying its challenging discretization needs, and outlining a plan for interaction with CEED. Working with all CEED thrusts, the liaisons will be ultimately responsible for ensuring the adoption of CEED technologies in each application as well as for demonstrating their benefits. The selected second wave applications and CEED liaisons are: Urban - liaison Aleks Obabko (ANL), ExaWind - liaison Paul Fischer (UIUC), and ExaAM - liaison Tzanio Kolev (LLNL). In addition to details and results from these efforts, in this document we report on other project-wide activities performed in Q1 of FY19, including: engagements with performing scaling on V100 GPUs, collaboration with ECP/MPICH, participation in ECP/ST’s E4S release, the Laghos-2.0 release, papers, presentations, and other outreach efforts.

97 MATHEMATICS AND COMPUTING↗

A Characteristic-Based Spectral Element Method for Moving-Domain Problems

Here, we introduce a characteristic-based numerical procedure for simulating incompressible flows in domains with moving boundaries. Our approach utilizes an operator-integration-factor splitting technique to help produce an efficient and stable numerical scheme. Using the spectral element method and an arbitrary Lagrangian-Eulerian formulation, we explore flows where the convective acceleration effects are non-negligible. Several examples, ranging from laminar to turbulent flows, are considered. Comparisons with a standard, semi-implicit time-stepping procedure illustrate the improved performance of our scheme.

97 MATHEMATICS AND COMPUTING↗

Performance tuning of CEED software and first wave apps

The goal of this milestone was the performance tuning of the CEED software and first wave apps. In this milestone, the CEED team developed optimization techniques and tuned for performance the CEED software. Specifically, the focus was on the following: (1) Fast finite element operator storage and evaluation using partial assembly/matrix-free algorithms that take advantage of tensor-product element structure; (2) Architecture optimizations, including the development of performant discretization libraries targeting heterogeneous systems with multi-core CPUs, GPUs, and/or many-core CPUs in single node, cluster, or large-scale ECP supercomputer configurations; and (3) Global kernels for finite element operators, including the development of algorithms and libraries for fast gather-scatter exchanges for action assembly on unstructured graphs and for components needed for global linear solvers. A main part of this milestone was the performance tuning of the CEED first-wave ECP applications. This included the ExaSMR application – Coupled Monte Carlo Neutronics and Fluid Flow Simulation of Small Modular Reactors (ORNL), and the MARBL application – Next-Gen Multi-physics Simulation Code (LLNL). The artifacts delivered include performance improvements in CEED’s 1st wave of applications, and tuned CEED software for various architectures through a number of backends, freely available in the CEED’s repository on GitHub. See the CEED website, http://ceed.exascaleproject.org and the CEED GitHub organization, http://github.com/ceed for more details. In addition to details and results from the above R&D efforts, in this document we are also reporting on other project-wide activities performed in Q4 of FY18 including: CEED’s second annual meeting, a successful minisymposium at the premier international conference on high-order methods (ICOSAHOM), making the CEED milestone reports publicly available, new Laghos release, and other outreach efforts.

97 MATHEMATICS AND COMPUTING↗

Quantum Sensing for High Energy Physics

Report of the first workshop to identify approaches and techniques in the domain of quantum sensing that can be utilized by future High Energy Physics applications to further the scientific goals of High Energy Physics.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗