Engineering Papers⌕ Search

Engineering topics

Fischer, Paul

Publications and source records attributed to Fischer, Paul.

At least 55 records · Page 3
All sourcesNASA NTRSDOE OSTI
DOE OSTI2019

OpenACC acceleration for the P N – P N - 2 algorithm in Nek5000

Due to its high performance and throughput capabilities, GPU-accelerated computing is becoming a popular technology in scientific computing, in particular using programming models such as CUDA and OpenACC. The main advantage with OpenACC is that it enables to simply port codes in their ‘‘original’’ form to GPU systems through compiler directives, thus allowing an incremental approach. An OpenACC implementation is applied to the CFD code Nek5000 for simulation of incompressible flows, based on the spectral-element method. The work follows up previous implementations and focuses now on the P N - P N-2 method for the spatial discretization of the Navier–Stokes equations. Performance results of the ported code show a speed-up of up to 3.1 on multi-GPU for a polynomial order N > 11.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Engage second wave ECP/CEED applications

The CEED co-design center aims to impact a wide range of ECP application teams through focused 1-to-1 and 1-to many interactions, facilitated by CEED application liaisons, ultimately targeted to provide easy-to-use discretization libraries for high-order FEM-based application simulations. The goal of this milestone was to extend our reach by identifying a second wave of ECP applications that can benefit from the efficient discretization algorithms and software developed in CEED. These applications were added to the list of the CEED first wave ECP applications (ExaSMR and MARBL), which continue to be main targets for our work. As part of the milestone, we also appointed liaisons on the CEED team that will be responsible for engaging each application, identifying its challenging discretization needs, and outlining a plan for interaction with CEED. Working with all CEED thrusts, the liaisons will be ultimately responsible for ensuring the adoption of CEED technologies in each application as well as for demonstrating their benefits. The selected second wave applications and CEED liaisons are: Urban - liaison Aleks Obabko (ANL), ExaWind - liaison Paul Fischer (UIUC), and ExaAM - liaison Tzanio Kolev (LLNL). In addition to details and results from these efforts, in this document we report on other project-wide activities performed in Q1 of FY19, including: engagements with performing scaling on V100 GPUs, collaboration with ECP/MPICH, participation in ECP/ST’s E4S release, the Laghos-2.0 release, papers, presentations, and other outreach efforts.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

A Characteristic-Based Spectral Element Method for Moving-Domain Problems

Here, we introduce a characteristic-based numerical procedure for simulating incompressible flows in domains with moving boundaries. Our approach utilizes an operator-integration-factor splitting technique to help produce an efficient and stable numerical scheme. Using the spectral element method and an arbitrary Lagrangian-Eulerian formulation, we explore flows where the convective acceleration effects are non-negligible. Several examples, ranging from laminar to turbulent flows, are considered. Comparisons with a standard, semi-implicit time-stepping procedure illustrate the improved performance of our scheme.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Performance tuning of CEED software and first wave apps

The goal of this milestone was the performance tuning of the CEED software and first wave apps. In this milestone, the CEED team developed optimization techniques and tuned for performance the CEED software. Specifically, the focus was on the following: (1) Fast finite element operator storage and evaluation using partial assembly/matrix-free algorithms that take advantage of tensor-product element structure; (2) Architecture optimizations, including the development of performant discretization libraries targeting heterogeneous systems with multi-core CPUs, GPUs, and/or many-core CPUs in single node, cluster, or large-scale ECP supercomputer configurations; and (3) Global kernels for finite element operators, including the development of algorithms and libraries for fast gather-scatter exchanges for action assembly on unstructured graphs and for components needed for global linear solvers. A main part of this milestone was the performance tuning of the CEED first-wave ECP applications. This included the ExaSMR application – Coupled Monte Carlo Neutronics and Fluid Flow Simulation of Small Modular Reactors (ORNL), and the MARBL application – Next-Gen Multi-physics Simulation Code (LLNL). The artifacts delivered include performance improvements in CEED’s 1st wave of applications, and tuned CEED software for various architectures through a number of backends, freely available in the CEED’s repository on GitHub. See the CEED website, http://ceed.exascaleproject.org and the CEED GitHub organization, http://github.com/ceed for more details. In addition to details and results from the above R&D efforts, in this document we are also reporting on other project-wide activities performed in Q4 of FY18 including: CEED’s second annual meeting, a successful minisymposium at the premier international conference on high-order methods (ICOSAHOM), making the CEED milestone reports publicly available, new Laghos release, and other outreach efforts.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Initial CEED API (CEED-MS10)

Key components of CEED software development involve fast finite element operator storage and evaluation, architecture optimizations, performant algorithms for all orders, global kernels for finite element operators, efficient use of the memory sub-system, optimal data locality and motion, enhanced scalability and parallelism, and fast tensor contractions. As ultimate goals of the project, CEED is exploring and identifying the best algorithms for the full range of discretizations and applying algorithmic and software development to support ECP applications’ needs. In this milestone, we initiated the development and implementation of a common CEED API consisting of two main components: low-level and high-level APIs. We accomplished the goal of having the low-level component ready for the release of CEED v1.0 in the next quarter, see CEED-MS13. In this report we also provide a highlight from CEED’s integration with the MARBL app and the corresponding Laghos miniapp. A main theme in this work has been the systematic transition of advanced algorithms, developed and tested in Laghos, to the large-scale multi-physics settings of the MARBL code. The low-level CEED API provides a set of FE kernels and components for writing new low-level kernels. Examples include: vector and sparse linear algebra, element matrix assembly over a batch of elements, partial assembly and action for efficient high-order operators like mass, diffusion, advection, etc. The main goal of the low-level API is to establish the basis for the high-level API. Also, identifying such low-level kernels and providing a reference implementation for them serves as the basis for collaboration with vendors and STs. Another goal for the low-level API is to make it easy for applications with their own discretization infrastructure, who do not want to switch directly to using the high-level CEED API, to still evaluate and use the core operations provided by the low-level API. One of the challenges with high-order methods is that a global sparse matrix is no longer a good representation of a high-order linear operator, both with respect to the FLOPs needed for its evaluation, as well as the memory transfer needed for a matvec. Thus, another important question we tried to address while developing the low-level API is: is it possible to abstract out in a natural and efficient way the different implementations and data structures that arise when dealing with various computational device types: CPUs, GPUs, etc. Fully answering this question is beyond the scope of this particular milestone, since that requires longer time as we develop and expand the devices we support. The future high-level API will be the frontend provided by CEED consisting of high-level abstractions such as meshes, finite element spaces and solutions, bilinear forms, solvers, etc. At this high-level, an important aspect of the API will be to provide a uniform abstraction layer for writing code in a portable manner. To achieve that, we plan to utilize the MPI programming model combined with a device abstraction layer (DAL) for finite element computations that we will develop. The DAL will serve as the bridge between the low- and the high-level APIs, allowing a variety of programming models to be used for the low-level kernel implementations. The software artifacts delivered as part of this milestone include an initial CEED API software implementation (libCEED) provided through the CEED website, http://ceed.exascaleproject.org and the CEED GitHub organization, http://github.com/ceed. In addition to details and results from the above R&D efforts, in this document we are also reporting on other project-wide activities performed in Q1 of FY18, including: the Nek5000 v17.0 MFEM v3.3.2 and PETSc v3.8 releases, collaborations with ECP/ST and SciDAC projects, the inaugural Nek5000 hackathon and various outreach activities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

CEED-MS8:Initial Integration of CEED Software in ECP Applications

Key components of CEED software development involve fast finite element operator storage and evaluation, architecture optimizations, performant algorithms for all orders, global kernels for finite element operators, efficient use of the memory sub-system, optimal data locality and motion, enhanced scalability and parallelism, and fast tensor contractions. As ultimate goals of the project, CEED is exploring and identifying the best algorithms for the full range of discretizations and applying algorithmic and software development to support ECP applications’ needs. In this milestone we report on the initial integration of CEED software in the selected first-wave ECP/CEED apps and the continued exploration of other ECP applications as second-wave (milestone CEED-MS23) and third-wave (milestone CEED-MS35) ECP/CEED application candidates. An extensive evaluation of various approaches for GPU acceleration of the CEED computational motives is also a major focus of the report, both in the settings of CEED’s Nekbone and Laghos miniapps, as well as versions of CEED’s bake-off problems (introduced in milestone CEED-MS6). The first-wave ECP applications targeted for coupling with CEED software include the ExaSMR and MARBL applications. We also report on near-term collaboration with the seed-funded Urban Systems project, as well as engagements with ACME, ExaAM, GEOS and the Application Assessment and Proxy App projects. The ExaSMR project involves coupled thermal-hydraulics/neutronics, with the latter based on Monte Carlo methods and the former based on high-order discretizations of the Navier-Stokes and energy equations for accurate simulation of turbulent transport. The project requires fast and efficient turbulence simulation capabilities that will scale to exascale platforms. Work within CEED will ensure that all aspects of the Nek5000 workflow scale to the target problem space, which involves > 10 10 degrees of freedom in typical production runs. CEED’s goal is to provide implementations that perform with maximum attainable efficiency and deliver low turn-around times. A deliverable for this milestone is beta-release of the GPU-based variant of Nek5000. MARBL is targeting high-energy-density physics problems, including single fluid multi-material hydrodynamics and radiation/magnetic diffusion simulation, with applications in inertial confined fusion, pulsed power experiments, and equation of state/material strength analysis. The code uses high-order methods based arbitrary Lagrangian-Eulerian, direct Eulerian, and unstructured adaptive mesh refinement. A deliverable for this milestone is collaboration with MARBL for improved performance based on partially-assembled operators. The Urban exascale project is planning to couple real-time data with high performance simulations to study particulate transport in cities. Nek5000 will be used for large eddy and RANS (Reynolds-averaged Navier-Stokes) simulations of turbulent transport in these domains. Areas where the CEED team is aiding the Urban Canyon group include creating high-order meshes for urban geometries and developing boundary conditions appropriate for simulations close to real-life urban conditions. The software artifacts delivered as part of this milestone include GPU-enabled versions of the Nekbone and Laghos miniapps, which are provided through the CEED website, http://ceed.exascaleproject.org and the CEED GitHub organization, http://github.com/ceed. In addition to details and results from the above R&D efforts, in this document we are also reporting on other project-wide activities performed in Q4 of FY17, including: the CEED first annual meeting and participation of CEED researchers in a variety of outreach activities.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

ECP Milestone Report: Identify initial kernels, bake-off problems (benchmarks) and miniapps (CEED-MS6)

The CEED co-design center is developing a number of kernels, bake-off / benchmark problems (BPs) and mini-applications (miniapps) that capture the unique requirements of high-order finite element algorithms and use them to influence standards, best practices and vendors to generate more portable, better performing code across diverse hardware designs. This is a main activity in CEED’s Hardware, Applications and Finite Element thrusts. CEED’s miniapps, a.k.a. CEEDlings, are small, standalone applications - surrogates for sub-components of interest from the selected first wave of applications (the MARBL/LLNLApp and ExaSMR projects). The BPs are even smaller standalone programs that solve a simple physical problem, including discretization, linear and/or nonlinear solve, and potentially a relatively small number of time steps. A BP may be a proxy/surrogate for part of a miniapp or another problem of general interest. In CEED, both miniapps and BPs will be used extensively in the project to evaluate and compare the performance of algorithms in specific contexts. The parts of the miniapps and BPs that play an important role in their overall performance will be identified and separated as standalone kernels. These kernels and their interaction inside BPs, miniapps and the actual applications, will be the main focus of our optimization efforts. All three standalone pieces of software (miniapps, BPs, and kernels) will serve as a basis for engaging and collaborating with hardware vendors and software technologies (ST) projects. These partners can start at the bottom of the hierarchy, e.g. with the kernels, and easily transition to BPs and miniapps as all three families will be produced in the same CEED environment. In this milestone we identified the initial kernels, bake-off problems and miniapps and delivered software and documentation for them through the new CEED website, http://ceed.exascaleproject.org. We specifically identified the Nekbone miniapp for the ExaSMR application and developed a new miniapp, Laghos, for the MARBL/LLNLApp application. We also formulated four initial benchmark problems (BP1- BP4) with clearly defined performance metrics and performed an initial bake-off comparison between the Nek5000 (spectral element) and MFEM (high-order finite element) technologies. As part of the milestone, we also engaged vendors (AMD) and ST projects (MPICH, STRUMPACK). In this document we are reporting some details and results from these R&D activities, as well as additional project-wide activities performed in Q3 of FY17, including: making CEED software available on GitHub, developing the project website, and the preparation of CEED’s first annual meeting.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

Brookhaven Lab Hosts Hackathon Focused on Graphics Processing Units (GPUs)

For five days in June 2017, Brookhaven Lab’s Computational Science Initiative hosted “Brookathon”—a programming workshop where teams of researchers, students, and software developers worked with mentors to accelerate scientific applications on GPUs for high-performance computing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2017

A performance analysis of ensemble averaging for high fidelity turbulence simulations at the strong scaling limit

We analyze the potential performance benefits of estimating expected quantities in large eddy simulations of turbulent flows using true ensembles rather than ergodic time averaging. Multiple realizations of the same flow are simulated in parallel, using slightly perturbed initial conditions to create unique instantaneous evolutions of the flow field. Each realization is then used to calculate statistical quantities. Provided each instance is sufficiently de-correlated, this approach potentially allows considerable reduction in the time to solution beyond the strong scaling limit for a given accuracy. This study focuses on the theory and implementation of the methodology in Nek5000, a massively parallel open-source spectral element code.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2017

Turbulent Flow Simulation at the Exascale: Opportunities and Challenges Workshop: August 4-5, 2015, Washington, D.C.

This report details the impact exascale will bring to turbulent-flow simulations in applied science and technology. The need for accurate simulation of turbulent flows is evident across the DOE applied-science and engineering portfolios, including combustion, plasma physics, nuclear-reactor physics, wind energy, and atmospheric science. The workshop brought together experts in turbulent-flow simulation, computational mathematics, and high-performance computing. Building upon previous ASCR workshops on exascale computing, participants defined a research agenda and path forward that will enable scientists and engineers to continually leverage, engage, and direct advances in computational systems on the path to exascale computing.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2016

Large-scale large eddy simulation of nuclear reactor flows: Issues and perspectives

Numerical simulation has been an intrinsic part of nuclear engineering research since its inception. In recent years a transition is occurring toward predictive, first-principle-based tools such as computational fluid dynamics. Even with the advent of petascale computing, however, such tools still have significant limitations. In the present work some of these issues, and in particular the presence of massive multiscale separation, are discussed, as well as some of the research conducted to mitigate them. Petascale simulations at high fidelity (large eddy simulation/direct numerical simulation) were conducted with the massively parallel spectral element code Nek5000 on a series of representative problems. These simulations shed light on the requirements of several types of simulation: (1) axial flow around fuel rods, with particular attention to wall effects; (2) natural convection in the primary vessel; and (3) flow in a rod bundle in the presence of spacing devices. Finally, the focus of the work presented here is on the lessons learned and the requirements to perform these simulations at exascale. Additional physical insight gained from these simulations is also emphasized.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2016

An MPI/OpenACC implementation of a high-order electromagnetics solver with GPUDirect communication

We present performance results and an analysis of a message passing interface (MPI)/OpenACC implementation of an electromagnetic solver based on a spectral-element discontinuous Galerkin discretization of the time-dependent Maxwell equations. The OpenACC implementation covers all solution routines, including a highly tuned element-by-element operator evaluation and a GPUDirect gather–scatter kernel to effect nearest neighbor flux exchanges. Modifications are designed to make effective use of vectorization, streaming, and data management. Performance results using up to 16,384 graphics processing units of the Cray XK7 supercomputer Titan show more than 2.5× speedup over central processing unit-only performance on the same number of nodes (262,144 MPI ranks) for problem sizes of up to 6.9 billion grid points. We discuss performance-enhancement strategies and the overall potential of GPU-based computing for this class of problems.

Computer Science↗
DOE OSTI2016

Nekbone performance on GPUs with OpenACC and CUDA Fortran implementations

We present a hybrid GPU implementation and performance analysis of Nekbone, which represents one of the core kernels of the incompressible Navier–Stokes solver Nek5000. The implementation is based on OpenACC and CUDA Fortran for local parallelization of the compute-intensive matrix–matrix multiplication part, which significantly minimizes the modification of the existing CPU code while extending the simulation capability of the code to GPU architectures. Our discussion includes the GPU results of OpenACC interoperating with CUDA Fortran and the gather–scatter operations with GPUDirect communication. We demonstrate performance of up to 552 Tflops on 16, 384 GPUs of the OLCF Cray XK7 Titan.

Computer Science↗
DOE OSTI2016

On the Strong Scaling of the Spectral Element Solver Nek5000 on Petascale Systems. In: EASC '16 Proceedings of the Exascale Applications and Software Conference 2016, Article No. 5

The present work is targeted at performing a strong scaling study of the high-order spectral element fluid dynamics solver Nek5000. Prior studies such as [5] indicated a recommendable metric for strong scalability from a theoretical viewpoint, which we test here extensively on three parallel machines with different performance characteristics and interconnect networks, namely Mira (IBM Blue Gene/Q), Beskow (Cray XC40) and Titan (Cray XK7). The test cases considered for the simulations correspond to a turbulent flow in a straight pipe at four different friction Reynolds numbers Reτ = 180, 360, 550 and 1000. Considering the linear model for parallel communication we quantify the machine characteristics in order to better assess the scaling behaviors of the code. Subsequently sampling and profiling tools are used to measure the computation and communication times over a large range of compute cores. We also study the effect of the two coarse grid solvers XXT and AMG on the computational time. Super-linear scaling due to a reduction in cache misses is observed on each computer. The strong scaling limit is attained for roughly 5000-10,000 degrees of freedom per core on Mira, 30,000 - 50,0000 on Beskow, with only a small impact of the problem size for both machines, and ranges between 10,000 and 220,000 depending on the problem size on Titan. This work aims at being a reference for Nek5000 users and also serves as a basis for potential issues to address as the community heads towards exascale supercomputers.

Offermans, Nicolas↗
DOE OSTI2015

OpenACC acceleration of the Nek5000 spectral element code

We present a case study of porting NekBone, a skeleton version of the Nek5000 code, to a parallel GPU-accelerated system. Nek5000 is a computational fluid dynamics code based on the spectral element method used for the simulation of incompressible flow. The original NekBone Fortran source code has been used as the base and enhanced by OpenACC directives. The profiling of NekBone provided an assessment of the suitability of the code for GPU systems, and indicated possible kernel optimizations. To port NekBone to GPU systems required little effort and a small number of additional lines of code (approximately one OpenACC directive per 1000 lines of code). The naïve implementation using OpenACC leads to little performance improvement: on a single node, from 16 Gflops obtained with the version without OpenACC, we reached 20 Gflops with the naïve OpenACC implementation. An optimized NekBone version leads to a 43 Gflop performance on a single node. In addition, we ported and optimized NekBone to parallel GPU systems, reaching a parallel efficiency of 79.9% on 1024 GPUs of the Titan XK7 supercomputer at the Oak Ridge National Laboratory.

Computer Science↗
DOE OSTI2015

Large-Scale Simulation of Nuclear Reactors: Issues and Perspectives

Numerical simulation has been an intrinsic part of nuclear engineering research since its inception. In recent years a transition is occurring toward predictive, first-principle-based tools such as computational fluid dynamics. Even with the advent of petascale computing, however, such tools still have significant limitations. In the present work some of these issues, and in particular the presence of massive multiscale separation, are discussed, as well as some of the research conducted to mitigate them. Petascale simulations at high fidelity (large eddy simulation/direct numerical simulation) were conducted with the massively parallel spectral element code Nek5000 on a series of representative problems. These simulations shed light on the requirements of several types of simulation: (1) axial flow around fuel rods, with particular attention to wall effects; (2) natural convection in the primary vessel; and (3) flow in a rod bundle in the presence of spacing devices. The focus of the work presented here is on the lessons learned and the requirements to perform these simulations at exascale. Additional physical insight gained from these simulations is also emphasized.

CFD↗
DOE OSTI2014

High-resolution coupled physics solvers for analysing fine-scale nuclear reactor design problems

An integrated multi-physics simulation capability for the design and analysis of current and future nuclear reactor models is being investigated, to tightly couple neutron transport and thermal-hydraulics physics under the SHARP framework. Over several years, high-fidelity, validated mono-physics solvers with proven scalability on petascale architectures have been developed independently. Based on a unified component-based architecture, these existing codes can be coupled with a mesh-data backplane and a flexible coupling-strategy-based driver suite to produce a viable tool for analysts. The goal of the SHARP framework is to perform fully resolved coupled physics analysis of a reactor on heterogeneous geometry, in order to reduce the overall numerical uncertainty while leveraging available computational resources. Finally, the coupling methodology and software interfaces of the framework are presented, along with verification studies on two representative fast sodium-cooled reactor demonstration problems to prove the usability of the SHARP framework.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗