Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “supercomputing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 631 records · Page 35
All sourcesNASA NTRSDOE OSTI
DOE OSTI2022

US Department of Energy, Office of Science High Performance Computing Facility Operational Assessment 2021: Oak Ridge Leadership Computing Facility

Oak Ridge National Laboratory’s (ORNL’s) Leadership Computing Facility (OLCF) continues to surpass its operational target goals of supporting users; delivering fast, reliable computational ecosystems; creating innovative solutions for high-performance computing (HPC) needs; contributing to the community to build the next generation HPC workforce, and managing risks, safety, and security associated with operating some of the most powerful computers in the world. The results can be seen in the cutting-edge science conducted by users and the praise from the research community. Calendar year (CY) 2021 saw continued excellence in research supported by the OLCF’s leadership-class computing resources, including Summit (the nation’s most powerful supercomputer), the global scratch file system Alpine, the Scalable Protected Infrastructure (SPI), the Exploratory Visualization Environment for Research in Science and Technology (EVEREST), and the archival mass-storage resource High-Performance Storage System (HPSS). While maintaining access and exceptional user support for Summit, the OLCF continued to make progress on the installation and deployment of Frontier, which will be the nation’s first exascale system when it comes online at the start of CY 2023. Users have already begun running and optimizing scientific codes on Crusher, the OLCF test and development system equipped with Frontier’s architecture. Throughout the year, the OLCF maintained a strong culture of operational excellence, including risk management, workplace safety, and cybersecurity. The OLCF’s rigorous risk management strategy anticipated and mitigated risks, and at this time there are no high-priority operational risks. Similarly, ORNL and the OLCF were committed to operating under the US Department of Energy’s (DOE’s) safety regulations that ensure a safe workplace. Technical staff tracked and monitored existing threats and vulnerabilities within the OLCF while continually developing tools and practices to enhance operations without increasing the facility’s risk. CY 2021 was filled with outstanding results and accomplishments, including a very high rating from users on overall satisfaction for the eighth consecutive year; a tremendous number of node hours delivered to 1,671 researchers on Summit; and the successful delivery of the allocation split of roughly 60%, 20%, and 20% of core-hours offered for the Innovative and Novel Computational Impact on Theory and Experiment (INCITE), Advanced Scientific Computing Research Leadership Computing Challenge (ALCC), and Director’s Discretionary (DD) programs, respectively (Section 2). COVID-19 research remained a focus in 2021, and the ALCC and DD programs allocated over 1 million Summit hours to the COVID-19 High Performance Computing Consortium. These accomplishments, coupled with the high utilization rates (i.e., overall and capability usage), represent the fulfillment of the promise of leadership class machines: efficient facilitation of leadership-class computational applications.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Can the United States Maintain Its Leadership in High-Performance Computing? - A report from the ASCAC Subcommittee on American Competitiveness and Innovation to the ASCR Office

The United States (U.S.) is no longer the unambiguous leader in the vitally important field of high performance computing (HPC). Japan, the European Union (EU), and China have fielded systems that are on par with our fastest supercomputers. The supply chain for everything from semiconductors to scientific software is globally distributed. Yet our economic future and security depend critically on our ability to innovate faster than our competitors, and the speed of innovation depends increasingly on large-scale computational science and engineering and thus HPC. How should the United States respond to this challenge? This report seeks to initiate a new and potentially transformative national discussion on this vital question. The Department of Energy’s (DOE) Advanced Scientific Computing Research (ASCR) program is well-positioned to make informed, targeted decisions about where the United States should cooperate and where it should compete in the global market for scientific exploration and discovery. By setting its sights on problems critical to our nation and the world, by establishing productive new collaborations, and by making strategic investments, ASCR can restore and maintain U.S. scientific leadership in the critical areas described in this report while strengthening our research infrastructure and training a large, diverse cohort of scientists. In doing so, ASCR and its scientists will pave the way for a secure and prosperous future for America. For more than 30 years, the ASCR program has provided the HPC and networking capabilities and expertise needed to support DOE’s mission to advance the national, economic, and energy security of the United States. The program now faces the challenge of developing and deploying the next generation of HPC systems and technologies, as well as supporting the application of HPC and artificial intelligence (AI) technologies to a wide range of scientific and engineering research problems. Through its research and development efforts, the ASCR program must also advance the state of the art in HPC and accelerate the pace of scientific discovery and technological innovation. Fulfilling this promise will require significantly increased investments, as well as innovative policies and programs. This subcommittee is aware that we are making recommendations and calls for action at a time when federal resources are limited. We understand that a wide range of competing priorities must be balanced by the nation’s leaders and that there is a need to leverage resources in new ways and seek efficiencies in facilities and operations. However, we must not let these realities limit our imagination or silence our advocacy. The ASCR program is a key part of the U.S. research infrastructure and an important component of economic growth and U.S. competitiveness. ASCR has a responsibility to pursue its mission, including advanced scientific computing, applications of AI technologies, and the required advanced research facilities, with determination and enthusiasm. To fulfill the scientific enterprise’s responsibility to the nation, the ASCR program must not only develop and publish a clear vision with an associated list of goals, priorities, and recommendations but also demonstrate scientific leadership by consistently securing long-term funding. This will allow the program to build on its achievements to date, to realize its ambitious vision, and to make lasting contributions to the field.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗
DOE OSTI2023

Computational Science at Los Alamos National Laboratory (Rev. 1) [Slides]

High Performance Computing (HPC) enables problem solving at the grandest scales, on the most complex challenges that face humanity. The problems often tackled by scientific computing are: Very complex; Cool; Using the fastest, most powerful supercomputers on the planet; Often important, impactful, effecting real lives.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Optimizing Performance on Trinity Utilizing Machine Learning, Proxy Applications and Scheduling Priorities

The sheer number of nodes continues to increase in today’s supercomputers, the first half of Trinity alone contains more than 9400 compute nodes. Since the speed of today’s clusters are limited by the slowest nodes, it more important than ever to identify slow nodes, improve their performance if it can be done, and assure minimal usage of slower nodes during performance critical runs. This is an ongoing maintenance task that occurs on a regular basis and, therefore, it is important to minimize the impact upon its users by assessing and addressing slow performing nodes and mitigating their consequences while minimizing down time. These issues can be solved, in large part, through a systematic application of fast running hardware assessment tests, the application of Machine Learning, and making use of performance data to increase efficiency of large clusters. Proxy applications utilizing both MPI and OpenMP were developed to produce data as a substitute for long runtime applications to evaluate node performance. Machine learning is applied to identify underperforming nodes, and policies are being discussed to both minimize the impact of underperforming nodes and increase the efficiency of the system. In this paper, I will describe the process used to produce quickly performing proxy tests, consider various methods to isolate the outliers, and produce ordered lists for use in scheduling to accomplish this task.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

2022 Operational Assessment Report - Argonne Leadership Computing Facility

This Operational Assessment Report describes how the Argonne Leadership Computing Facility (ALCF) met or exceeded every one of its goals for the calendar year (CY) 2022. In CY 2022, ALCF operated Theta, an Intel-based Cray XC40 system (11.7 petaflops) augmented with 24 NVIDIA DGX A100-based nodes (3.9 petaflops), that supports diverse workloads, integrating data analytics with artificial intelligence (AI) training and learning in a single platform; and Polaris, a 44-petaflop AMD and NVIDIA-based HPE Apollo 6500 Gen10+ system that provides a powerful new platform to prepare applications and workloads for Aurora, Argonne National Laboratory’s (Argonne’s) upcoming Intel-Hewlett Packard Enterprise (HPE) exascale supercomputer.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

ESnet Requirements Review Program Through the IRI Lens: A Meta-Analysis of Workflow Patterns Across DOE Office of Science Programs (Final Report)

The Department of Energy (DOE) ensures America’s security and prosperity by addressing its energy, environmental, and nuclear challenges through transformative science and technology solutions. The DOE’s Office of Science (SC) delivers groundbreaking scientific discoveries and major scientific tools that transform our understanding of nature and advance the energy, economic, and national security of the United States. The SC’s programs advance DOE mission science across a wide range of disciplines and have developed the research infrastructure needed to remain at the forefront of scientific discovery. The DOE SC’s world-class research infrastructure — exemplified by the 28 SC scientific user facilities — provides the research community with premier observational, experimental, computational, and network capabilities. Each user facility is designed to provide unique capabilities to advance core DOE mission science for its sponsor SC program and to stimulate a rich discovery and innovation ecosystem. Research communities gather and flourish around each user facility, bringing together diverse perspectives. A hallmark of many facilities is the large population of students, postdoctoral researchers, and early-career scientists who contribute as full-fledged users. These facility staff and users collaborate over years to devise new approaches to utilizing the user facility’s core capabilities. The history of the SC user facilities has many examples of wildly inventive researchers challenging operational orthodoxy to pioneer new vistas of discovery; for example, the use of the synchrotron X-ray light sources for study of proteins and other large biological molecules. This continual reinvention of the practice of science — as users and staff forge novel approaches expressed in research workflows — unlocks new discoveries and propels scientific progress. Within this research ecosystem, the high-performance computing (HPC) and networking user facilities stewarded by SC’s Advanced Scientific Computing Research (ASCR) program play a dynamic cross-cutting role, enabling complex workflows demanding high performance data, networking, and computing solutions. The DOE SC’s three HPC user facilities and the Energy Sciences Network (ESnet) high-performance research network serve all of the SC’s programs as well as the global research community. Argonne Leadership Computing Facility (ALCF), the National Energy Research Scientific Computing Center (NERSC), and Oak Ridge Leadership Computing Facility (OLCF) conceive, build, and provide access to a range of supercomputing, advanced computing, and large-scale data-infrastructure platforms, while ESnet interconnects DOE SC research infrastructure and enables seamless exchange of scientific data. All four facilities operate testbeds to expand the frontiers of computing and networking research. Together, the ASCR facilities enterprise seeks to understand and meet the needs and requirements across SC and DOE domain science programs and priority efforts, highlighted by the formal requirements reviews (RRs) methodology. In recent years, the research communities around the SC user facilities have begun experimenting with and demanding solutions integrated with HPC and data infrastructure. This rise of integrated-science approaches is documented in many community and high-level government reports. At the dawn of the era of exascale science and the acceleration of artificial intelligence (AI) innovation, there is a broad need for integrated computational, data, and networking solutions. In response to these drivers, DOE has developed a vision for an Integrated Research Infrastructure (IRI): To empower researchers to meld DOE’s world-class research tools, infrastructure, and user facilities seamlessly and securely in novel ways to radically accelerate discovery and innovation.

42 ENGINEERING↗
DOE OSTI2023

An Implicit Approach to Phase Field Modeling of Solidification for Additively Manufactured Alloys [Slides]

We are leveraging modern algorithms and computational science to provide a route to predictive simulation of microstructure evolution on emerging exascale architectures. We are utilizing the fastest supercomputers in the world for modeling and simulation of microstructure evolution for generation of data under AM conditions. Solidification conditions in AM can be tailored for the reliable design of materials to specific performance requirements. Developing computational tools to further characterize alloys and correlate the processing-structure-properties-performance (PSPP) relationship.

36 MATERIALS SCIENCE↗
DOE OSTI2023

Parallel Algebraic Multigrid for Fusion and Higher-Order PDEs

Multigrid methods play a key role in large-scale scientific simulation because they are among the fastest and most scalable approaches for solving the underlying sparse linear systems of equations that arise from a wide array of Partial Differential Equation (PDE) discretizations. Algebraic multigrid (AMG) is a special type of multigrid method that depends only on the description of the linear system, giving it better portability and broader applicability than geometric multigrid, as it requires no explicit knowledge of the problem geometry. Even though these methods are widely used today, there are still applications where further development is needed. In this report, we focus on PDEs with higher-order terms (e.g., fourth order), concentrating on a PDE that arises in tokamak edge plasma simulations (a tokamak is a machine that confines a plasma using magnetic fields and is believed to be the leading plasma confinement concept for future fusion power plants). General multigrid relaxes a linear system on coarser grids and reverses this process with interpolation, but standard AMG methods struggle with the aforementioned higher-order PDEs. We investigate cyclic coarsening and interpolation heuristics, as well as new iterative approximation methods of refining the solution at each grid to improve the existing multigrid approach. To this end, we ensure that these techniques are transferable to a parallelized setting with LLNL’s supercomputers.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

User Manual - HydraGNN: Distributed PyTorch Implementation of Multi-Headed Graph Convolutional Neural Networks

This document serves as user manual for HydraGNN, a scalable graph neural network (GNN) architecture that allows for a simultaneous prediction of multiple target properties using multi-task learning (MTL). The HydraGNN architecture is constructed by successive superposition of three different sets of layers. The first set is made of message-passing layers to exchange information across nodes in the graph and use this to update the nodal features. The second set is made of global pooling layers that aggregate information from all the nodes in the graph and map it into a scalar, and is needed only for global target properties that are related to the entire graph. The third set of layers is dedicated to the implementation of MTL, which is enabled by forking of the architecture into separate heads, each one of them dedicated to the predictive task of one specific target property. Through an object-oriented programming paradigm, HydraGNN is templated over different message-passing policies, which allows for a user-friendly hyperparameter study to assess the sensitivity of the predictive performance of the HydraGNN architecture on a specific dataset with respect to the choice of the message-passing policy. The object-oriented paradigm used by HydraGNN also allows for a user-friendly inclusion of newly developed message passing policies within the existing framework. HydraGNN supports distributed computing capabilities for scalable data reading and scalable training on leadership-class supercomputers.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Working with Multiple MPIs: Overcoming ABI Incompatibility

Scientific applications rely on third-party libraries, which may have multiple implementations. While sharing an application programming interface (API), many of these implementations do not have a shared application binary interface (ABI) and require recompiling. Recompiling can be a long and complex process and sometimes not even an option when the application is shipped binary only. ABI incompatibility strikes at the heart of portability, productivity, and performance by (1) impeding application execution across different HPC and Cloud systems; (2) adding developer hours rebuilding an application; and (3) not taking advantage of host-optimized libraries. This tutorial teaches attendees a portable way to address ABI incompatibility in MPI using the Wi4MPI library. Wi4MPI translates the ABI dynamically from the MPI library used to build the application to a different MPI library available at run time. With Wi4MPI, HPC practitioners can break the portability barrier imposed by ABI incompatibility, potentially increase performance, and increase user productivity. The tutorial is broken down in three components: (1) Understanding ABI compatibility in MPI; (2) Translating MPI libraries dynamically; and (3) Applying dynamic translation to key use cases in HPC, including Containers. If you use more than one MPI library or supercomputer, this tutorial is for you.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Surface Roughness Effects from Additive Manufacturing in High Efficiency Gas Turbine Combustion Systems

In the past, clean combustion systems were characterized by high system development costs due to growing complexity and escalating manufacturing costs from conventional manufacturing processes. As a result, high efficiency concepts are difficult to design and more difficult to be cost-effectively manufactured. In recent years, with the introduction of additive manufacturing (AM) technologies, the rapid prototyping and mass production processes of clean combustion systems are promising to be significantly simplified with significant reduction in terms of engine manufacturing cost and engine energy cost. Compared with conventional manufacturing built parts, the AM process enabled a simpler design to be adopted for the nozzle, reducing the number of required braze and weld joints from twenty-five to just five. The resulting nozzle was 25% lighter and five times more durable and contributed to a 15% reduction in fuel burn in comparison with the previous model produced. Applying these improvements to a fleet of 5,000 turbofan engines at 150 kN thrust would result in fuel cost savings of $ \$ $6B annually (at $ \$ $5/gallon Jet-A fuel price, 700 gallon/hour fuel consumption rate, and 2,300 operational hours per year), and reduce CO2 emissions by more than 25 million tons per year. These advances support core Department of Energy (DOE) missions in energy efficiency, improving productivity, and environmental sustainability. Maximizing the benefit of these new capabilities will require high prediction capability of high speed turbulent flow with wall modeled Large Eddy Simulation (LES). GE Aviation maintains that advanced simulation technology and supercomputing is required in order to provide the appropriate boundary conditions to realize the maximum potential of AM and Ceramic Matrix Composite to reduce cooling flow, a first order penalty on the Brayton cycle. The algorithm developed using the ANSYS/Fluent software would provide the foundation for entirely new avenues of research and development with potential multi-billion dollar impact to the US economy, and significant reduction in carbon based emissions across the aerospace and power generation industries. The formulation of this iWLES (integral wall model for LES) is generic and allows to capture the changes in flow dynamics that have a significant impact on the wall bounded flow characteristics, such as swirler effective area, bulk swirl number, local pressure distribution, exit velocity profile, and turbulence kinetic energy profile. A periodic channel flow with rough flat plate is simulated using LES (Wall-Adapting Local Eddy-viscosity) model to verify the implementation of iWLES in the Fluent User Defined Function (UDF). As the flow fields are highly sensitive due to surface roughness of the wall bounded flows in the combustion systems, there is significant potential to conduct further LES studies focusing on the turbulence boundary layer interaction. Accurately capturing near wall physics will elucidate the impact of rough surfaces on combustor flow and aero-thermal interactions.

42 ENGINEERING↗
DOE OSTI2023

Surface Roughness Effects from Additive Manufacturing in High Efficiency Gas Turbine Combustion Systems

In the past, clean combustion systems were characterized by high system development costs due to growing complexity and escalating manufacturing costs from conventional manufacturing processes. As a result, high efficiency concepts are difficult to design and more difficult to be cost-effectively manufactured. In recent years, with the introduction of additive manufacturing (AM) technologies, the rapid prototyping and mass production processes of clean combustion systems are promising to be significantly simplified with significant reduction in terms of engine manufacturing cost and engine energy cost. Compared with conventional manufacturing built parts, the AM process enabled a simpler design to be adopted for the nozzle, reducing the number of required braze and weld joints from twenty-five to just five. The resulting nozzle was 25% lighter and five times more durable and contributed to a 15% reduction in fuel burn in comparison with the previous model produced. Applying these improvements to a fleet of 5,000 turbofan engines at 150 kN thrust would result in fuel cost savings of $6B annually (at $5/gallon Jet-A fuel price, 700 gallon/hour fuel consumption rate, and 2,300 operational hours per year),and reduce CO2 emissions by more than 25 million tons per year. These advances support core Department of Energy (DOE)missions in energy efficiency, improving productivity, and environmental sustainability. Maximizing the benefit of these new capabilities will require high prediction capability of high speed turbulent flow with wall modeled Large Eddy Simulation (LES). GE Aviation maintains that advanced simulation technology and supercomputing is required in order to provide the appropriate boundary conditions to realize the maximum potential of AM and Ceramic Matrix Composite to reduce cooling flow, a first order penalty on the Brayton cycle. The algorithm developed using the ANSYS/Fluent software would provide the foundation for entirely new avenues of research and development with potential multi-billion dollar impact to the US economy, and significant reduction in carbon based emissions across the aerospace and power generation industries. The formulation of this iWLES (integral wall model for LES) is generic and allows to capture the changes in flow dynamics that have a significant impact on the wall bounded flow characteristics, such as swirler effective area, bulk swirl number, local pressure distribution, exit velocity profile, and turbulence kinetic energy profile. A periodic channel flow with rough flat plate is simulated using LES (Wall-Adapting Local Eddy-viscosity) model to verify the implementation of iWLES in the Fluent User Defined Function (UDF). As the flow fields are highly sensitive due to surface roughness of the wall bounded flows in the combustion systems, there is significant potential to conduct further LES studies focusing on the turbulence boundary layer interaction. Accurately capturing near wall physics will elucidate the impact of rough surfaces on combustor flow and aero-thermal interactions.

99 GENERAL AND MISCELLANEOUS↗
DOE OSTI2022

Science &Technology Review (September 2022)

Computer modeling is essential to scientific research. Models simulate natural phenomena to aid scientists in understanding their underlying principles. While the most complex models running on supercomputers may contain millions of lines of code and generate billions of data points, models never simulate reality perfectly. Experiments—in contrast—have been fundamental to the study of natural phenomena from science’s earliest days. However, some of today’s complex experiments generate too much data for the human mind to interpret.

42 ENGINEERING↗
DOE OSTI2024

Celeritas R&D Report: Accelerating Geant4

Celeritas is a new Monte Carlo (MC) detector simulation code designed for computationally intensive applications on high-performance heterogeneous architectures. In the past two years Celeritas has advanced from prototyping a Graphics Processing Unit (GPU)-based single physics model in infinite medium to implementing a full set of electromagnetic (EM) physics processes in complex geometries. The current release of Celeritas, version 0.4, has incorporated full device-based navigation, an event loop in the presence of magnetic fields, and detector hit scoring. New functionality incorporates a scheduler to offload electromagnetic physics to the GPU within a Geant4-driven simulation, enabling straightforward integration of Celeritas into the high energy physics (HEP) experimental frameworks CMSSW and ATLAS FullSimLight. On the Perlmutter supercomputer, Celeritas performs EM physics between 3× and 18× faster using the machine’s Nvidia GPUs compared to using only CPUs, corresponding to an electrical power efficiency up to a factor of 5. When running a multithreaded Geant4 ATLAS test beam application with full hadronic physics, using Celeritas to accelerate the EM physics results in an overall simulation speedup of 1.7–2.2× on GPU and 1.2× on CPU. In a CMS test application using tt¯ events and the prototype Run 4 configuration, compared to Geant4 CPU, Celeritas with a Nvidia A100 improves overall throughput up to a factor of 2.7× but cannot be efficiently shared with more than 8 cores.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Go with the Radflow: Thermal radiation experiments in the high energy-density regime [Slides]

Los Alamos National Laboratory is a nuclear weapons laboratory supporting our nation's defense. In support of the mission is a high energy-density physics program in which we design and execute experiments to study radiation-hydrodynamics phenomena and improve the predictive capability of our large-scale multi-physics software codes on our supercomputers. The Radflow Project is maturing a unique spectroscopic measurement that yields a spatially dependent supersonic radiation wavefront profile. The spectroscopic measurement is much more constraining than the typically used radiography, which measures the density changes due to the shock as the radiation wave goes subsonic. Recently, the spectroscopic measurement has been applied to advanced targets with a detailed heterogeneity representing a single realization of a stochastic medium. The Radflow Project also conducts opacity experiments to explain the current conundrum of the experimental iron opacity differing from theory, which is important in modeling the sun. More than that, though, we need to be able to answer these science questions and be able to validate our theory and codes.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Establishing capabilities for quantum computing and simulations for energy applications

Quantum information science (QIS) is creating potential transformative opportunities to exploit intricate quantum mechanical phenomena in new ways for obtaining and processing information to advance many areas of science and engineering. Since the National Quantum Initiative Act was signed into law in 2018, developing QIS capability and competency is one of the most urgent tasks of DOE to make sure the US win the quantum race. The QIS contains four pillars: quantum computing, quantum simulations, quantum sensing, and quantum networking. To apply QIS in energy related applications, the key is to develop the capability of quantum computing & simulation tools. In this project, we propose to develop the capability of quantum computing and simulations at NETL to target fossil energy related problems. We will install a simulator (e. g. IBM qiskit) on the NETL supercomputer to simulate the environments of quantum computer. Based on the current available quantum algorithms for quantum chemistry, we will develop quantum computing codes to perform simulation which will focus on fossil energy sector challenges, including CO2 capture & conversion, sensing, and fuel conversion. We then will seek opportunities to run our codes on real quantum computers (such as IBM-Q, google Sycamore, etc.). Through this project, the new capability of quantum computing and simulations will be established at NETL. In addition, the NETL workforce in this area will be trained ready to conduct more complicated tasks in line with NETL missions to enhance the nation’s energy foundation.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2024

Developing And Scaling an OpenFOAM Model to Study Turbulent Flow in a HFIR Coolant Channel

Improving the understanding of how computational fluid dynamics (CFD) direct numerical simulations (DNS) of flows in the High Flux Isotope Reactor (HFIR) perform when run in parallel using the high performance computing (HPC) platform Summit at the Oak Ridge Leadership Computing Facility (OLCF) is of particular importance to boost the computational tools used to support HFIR conversion to low enriched fuel (LEU). Evaluation of scaling performance was driven by the increasing importance of graphics processing unit (GPU) usage in HPC, which is becoming the standard for modern supercomputers such as Summit. The desired results are to obtain a strong positive correlation between the computational resources dedicated to a problem and the relative speed-up of the simulation in comparison to a benchmark. This capability will allow substantially improvement in HFIR flow analytical capabilities, specifically when predicting turbulence properties at high Reynolds numbers. The study leverages previous simulation results performed with code PHASTA (finite element) on HPC platforms Cori (NERSC) and Theta (ALCF) [1] with computing options provided in the computing platform OpenFOAM (finite volume) at OLCF. Transitioning from PHASTA to OpenFOAM will (1) eliminate dependence on third-party software for mesh generation and manipulation, (2) reduce resource needs by employing modern architectures, and (3) build expertise for future modeling of HFIR-specific problems like heat transfer in involute geometry, entrance effects, flow structure in channel corners, and so on—all important issues when defining the available thermal margins in the transition to LEU. CPUs and GPUs differ significantly in their architecture and utilization, as discussed in the literature [2]. The most important differences are in the approach to computations and their memory. A single GPU contains a large quantity of cores, enabling it to perform with a much higher throughput than a CPU, but execution requires a different approach. GPU codes execute instructions using the Single-Instruction Multiple-Thread (SIMT) approach in which a single instruction is used for groups of threads called warps. A warp typically consists of 32 threads which must execute the same set of instructions, although on separate threads. Alternately, a CPU has far fewer cores that are much more flexible in their operation, excelling at quickly performing more complex serial computations. This is why GPUs have greater throughput when properly utilized. The second important difference is seen when comparing their memory spaces. Limited memory allocations and CPU–GPU communications cause a significant bottleneck in GPU-accelerated programs. Further study was required to properly take advantage of GPU resources. A comprehensive analysis of code performance and the model-specific features of turbulence constitutes the core of this work. In this study, a DNS simulation of HFIR channel turbulence was performed with the finite volume CFD code OpenFOAM v2112 and CUDA v11.0 on Red Hat Enterprise Linux v8.2. The OpenFOAM installation had AMGx integrated to enable GPU acceleration and utilizes the PETSc4FOAM library. The computational resources and the problem size were scaled on CPU and CPU + GPU architectures to gain a better understanding of the performance of a DNS problem on modern computing hardware. The study aimed to analyze the scaling of the code exclusively on CPUs and then to examine the scaling of the codes with GPU acceleration enabled. Scaling studies included CPU and GPU acceleration on a mesh of varying resolution to analyze the impact of problem size relative to computational resources. In the course of preparing the GPU configuration on Summit, mainly using the AMGX solvers, difficulties were encountered stemming from constant changes resulting from extensive ongoing development activities and the changing environment. This resulted in the inability to complete the GPU portion of the work. The code was compiled and tested, but production runs to assess acceleration were not performed because the used discretional compute time allocation expired as year-end approached. The Summit HPC platform is scheduled for decommissioning in 2024, making it unattractive for future use with Nvidia-based GPUs. Therefore, the work will be moved onto NERSC machines in FY24. An application was prepared and submitted, and sufficient node-hours were awarded to continue the research in the next calendar year. This report summarizes work performed thus far, which mostly focused on CPU OpenFOAM computing.

21 SPECIFIC NUCLEAR REACTORS AND ASSOCIATED PLANTS↗