Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “high performance parallel computing”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,405 records · Page 78
All sourcesNASA NTRSDOE OSTI
DOE OSTI2016

Measurements of file transfer rates over dedicated long-haul connections

Wide-area file transfers are an integral part of several High-Performance Computing (HPC) scenarios. Dedicated network connections with high capacity, low loss rate and low competing traffic, are increasingly being provisioned over current HPC infrastructures to support such transfers. To gain insights into these file transfers, we collected transfer rate measurements for Lustre and xfs file systems between dedicated multi-core servers over emulated 10 Gbps connections with round trip times (rtt) in 0-366 ms range. Memory transfer throughput over these connections is measured using iperf, and file IO throughput on host systems is measured using xddprof. We consider two file system configurations: Lustre over IB network and xfs over SSD connected to PCI bus. Files are transferred using xdd across these connections, and the transfer rates are measured, which indicate the need to jointly optimize the connection and host file IO parameters to achieve peak transfer rates. In particular, these measurements indicate that (i) peak file transfer rate is lower than peak connection and host IO throughput, in some cases by as much as 50% or lower, (ii) xdd request sizes that achieve peak throughput for host file IO do not necessarily lead to peak file transfer rates, and (iii) parallelism in host IO and TCP transport does not always improve the file transfer rates.

Rao, Nageswara S↗
DOE OSTI2026

Computational modeling of graphite degradation in molten salt reactors: Role of infiltration

Molten salt reactors (MSRs) often employ graphite as a moderator and reflector. An important challenge for deploying graphite in these reactors is that, due to limited experimental data, our understanding of graphite’s structural integrity in molten salt environments remains incomplete. Here, this study addresses heat generation from fuel-bearing salt that has infiltrated open pores in the graphite, driven primarily by pressure differentials. This is one of multiple identified physical and chemical mechanisms through which molten salt could potentially degrade graphite. Thermally driven stresses are quantified using the Molten-Salt Reactor Experiment (MSRE) graphite moderator elements as a case study. Finite element simulations predict stress distributions at varying infiltration levels, indicating that thermal stresses increase with higher infiltration. Rare-event simulations using the parallel subset simulation framework identify the combinations and corresponding ranges of input parameters that lead to stresses above a specified threshold. In particular, combinations involving high infiltration amounts, high power density, and low thermal conductivity tend to induce the highest stresses. Under the inputs and assumptions considered in this work, the magnitudes of the thermally driven stresses are quite low, with a very low likelihood of causing failure due to exceeding the graphite’s tensile strength. Additionally, rare-event simulations were performed for two more scenarios: a scaled-up moderator geometry and a localized hotspot in the original geometry. Both cases resulted in increased susceptibility to failure, though not to a detrimental extent. Furthermore, the combined effects of irradiation and infiltration-induced thermal stresses were evaluated. The results showed that thermal stresses from infiltration were negligible compared to those caused by irradiation. The findings of such a study are inherently component-specific, but the methodology presented here could be used for similar assessments of salt-infiltration effects in other graphite components.

36 - MATERIALS SCIENCE↗
DOE OSTI2013

Materials Design and Discovery: Catalysis and Energy Storage (Mira Early Science Program Final Technical Report): ALCF-2 Early Science Program Technical Report

The investigation and design of new classes of materials for energy and catalysis requires a multi-facetted approach to simulation. Multiple methods are needed to study materials on the length scale 0.1 nm - 10 nm. For simulations where the atomic (and electronic) degrees of freedom are relevant, the methods of choice in the surface science, condensed matter physics, and material science communities are classical molecular dynamics (CMD), Density Functional Theory (DFT), and quantum Monte Carlo (QMC). The original scope for this Early Science Program (ESP) project was to perform fast-accurate DFT calculations on materials for energy and catalysis using the GPAW code on Blue Gene/Q. The types of calculations included significantly reduced time-to-solution on systems sizes accessible on Blue Gene/P (~10, 0000 valence electrons), but also systems which were were at least a factor of two larger (~20, 000 valence electrons). GPAW is a real-space DFT code using the projector augmented wave (PAW) method. DFT calculations on Blue Gene/P were executed on over >100, 000 cores using GPAW; thus it was consider a success on Mira’s predecessor system, Intrepid. One of the co-PIs (NAR), determined that the work necessary to allow the GPAW code for these aforementioned types of calculations could not be accomplish within the time frame of the ESP. This was not simply due to human time required to implement OpenMP parallelism, but also from intrinsic algorithmic limitations in supporting libraries, most notably ScaLAPACK. Additionally, the return-on-investment on $\mathcal{O}(N^3)$ DFT code has become some what tenable at best. NAR argues that what is really needed in preparation for exascale computing and to enable high-fidelity materials research is robust reduced scaling, $\mathcal{O}$(N) or $\mathcal{O}$(N)log(N), DFT approaches. Thus, in agreement with the other co-PIs, the decision was made to purse QMC as a complimentary method on Mira since it would be valuable for the scientific community and could easily leverage the massive parallelism that would be provided by the Blue Gene/Q. We note that the two other atomic-scale methods mentioned here, CMD and DFT, are being pursued by other ESP projects on Mira. The remainder of this report will focus on our progress on QMC.

Benali, Anouar↗
DOE OSTI2023

Fission with Exotic Nuclei (Abbreviated Report)

Nuclear fission is a key mechanism involved in the synthesis of heavy elements in the Cosmos and is the primary explanation for the stability of superheavy elements. Nevertheless, our knowledge of fission remains extremely fragmented. Most experiments have been conducted only on a tiny number of stable actinide nuclei and are often incomplete, leading to gaps in our basic understanding of the process. For many radioactive isotopes, basic fission data such as the charge or mass distribution of the fragments is unknown. These gaps cannot always be filled by simulation alone. Common fission models contain too many free parameters and lack predictive power. In contrast, the fundamental theory of fission under development at LLNL is much more predictive, but its current computational cost is too high to be used extensively for data evaluations. A unique window of opportunity to resolve these limitations has recently opened: the U.S. nuclear science community is ramping up major experimental programs at the Facility for Rare Isotope Beams (FRIB, the DOE flagship facility in low-energy nuclear science), and techniques from machine learning have shown great potential to simplify the use of a fundamental, quantum-mechanical theory of fission. This project has two components. On the experimental side, we acquired and deployed at the HIGS facility a new dual Frisch-Grid ionization chamber to measure correlated fragment-mass, kinetic energy, and angular distributions of fission fragments from induced fission. This new device was used to perform measurements of charge, mass and total kinetic energy of fission fragments in the photofission of 238 U and eight gamma-ray beam energies between 6.2 and 13 MeV, which allowed extracting high-precision independent yields for this reaction. The device was also used to perform measurements of the same quantities in the neutron-induced fission of 234 U with monoenergetic beams of energy between 5 and 8 MeV. In parallel, we collaborated with a team at Commissariat à l’énergie atomique et aux énergies alternatives (CEA) to perform a series of measurements of fission yields in inverse kinematics for the two isotopes of 236 U and 240 Pu. The experiment took place at the Grand Accélérateur National d’Ions Lourds in France in June 2023. The deployment of the VAMOS spectrometer with a new array called PISTA allowed determining the excitation energy of the fissioning system within 1 Mega-electronvolts. The second component of the project involved using deep neural networks to build fast and reliable emulators of our current fission models. In an invited paper published in Frontier in Physics, we showed that autoencoders could successfully compress nuclear wavefunctions in nuclear density functional theory. We achieved a dimensionality reduction of the order of two orders of magnitude while keeping the error in the total energy to less than 0.01%. In a second paper submitted to Physical Review Letters in June 2023 with our collaborators at CEA, we showed that variational autoencoders can learn the collective degrees of freedom driving the fission process.

73 NUCLEAR PHYSICS AND RADIATION PHYSICS↗
DOE OSTI2024

Enabling high-throughput enzyme discovery and engineering with a low-cost, robot-assisted pipeline

Abstract As genomic databases expand and artificial intelligence tools advance, there is a growing demand for efficient characterization of large numbers of proteins. To this end, here we describe a generalizable pipeline for high-throughput protein purification using small-scale expression in E. coli and an affordable liquid-handling robot. This low-cost platform enables the purification of 96 proteins in parallel with minimal waste and is scalable for processing hundreds of proteins weekly per user. We demonstrate the performance of this method with the expression and purification of the leading poly(ethylene terephthalate) hydrolases reported in the literature. Replicate experiments demonstrated reproducibility and enzyme purity and yields (up to 400 µg) sufficient for comprehensive analyses of both thermostability and activity, generating a standardized benchmark dataset for comparing these plastic-degrading enzymes. The cost-effectiveness and ease of implementation of this platform render it broadly applicable to diverse protein characterization challenges in the biological sciences.

36 MATERIALS SCIENCE↗
DOE OSTI2024

Computational Analysis of Hydraulic Efficiency of Michigan DOT Cover C

Drainage structures are used to capture stormwater runoff in streets and highways in urban environments. These drainage structures, which typically consist of catch basins with grates, inlets, or combination grates/inlets, collect stormwater runoff and discharge through buried conveyance systems. They are strategically placed for public safety in curb and gutter systems to provide efficient drainage of water from roadways and thus reduce the risk of hydroplaning. The performance of drainage structures is measured in terms of hydraulic efficiency, which is defined as the percentage of flow captured by the basin as compared to the total flow drainage to the structure. Understanding of the performance of these drainage structures helps designers properly space inlets to promote an economic design that ensures the safety of the traveling public. The current design methodology used to determine drainage structure follows guidelines established in the current Michigan Department of Transportation (MDOT) Drainage Manual (2006). The guidelines in the MDOT Drainage Manual were modeled after the Federal Highway Administration’s (FHWA) Hydraulic Engineering Circular 22 “Urban Drainage Design” (HEC-22). HEC-22 includes empirically derived equations to calculate the interception capacity of drainage structures for several commonly used grate configurations, such as the parallel bar, curved vane and tilt bar grates, which are based on a research study performed by Burgi et al. in the 1970s. MDOT uses several drainage structures to capture runoff that are detailed as Standard Plans. Many of these drainage structures utilize sinusoidal type grates that are not described in HEC-22. Physical modeling of these structures has been limited, posing the need to have them analyzed to verify their capture efficiency. Current MDOT practice is to assume a similar sized reticuline grate, as described in HEC-22, for capture efficiencies. Until recently, evaluating the hydraulic performance of drainage structures was limited to physical modeling in a hydraulics laboratory. With advances in engineering software and computing power, computational fluid dynamics (CFD) modeling has become a more cost-effective alternative. The Federal Highway Administration (FHWA) provides states the option to evaluate their drainage structures using CFD through the Transportation Pooled Fund Program. This study, “Computational Analysis of Hydraulic Efficiency of Michigan DOT Cover C,” was carried out using the pooled fund. MDOT’s Cover C was chosen as the first test candidate, given its similar sinusoidal pattern to other MDOT grates, but it is typically used for high-volume, higher speed applications. A similar version, Cover CX, is used on interstate highways but does not have traverse bars for bicycle safety. Additional grates may be considered for evaluation in the future.

42 ENGINEERING↗
DOE OSTI2016

Integration Of PanDA Workload Management System With Supercomputers for ATLAS and Data Intensive Science

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe, and were recently credited for the discovery of a Higgs boson. ATLAS, one of the largest collaborations ever assembled in the sciences, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data centers are physically scattered all over the world. While PanDA currently uses more than 250,000 cores with a peak performance of 0.3 petaFLOPS, LHC data taking runs require more resources than Grid computing can possibly provide. To alleviate these challenges, LHC experiments are engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. We will describe a project aimed at integration of PanDA WMS with supercomputers in United States, Europe and Russia (in particular with Titan supercomputer at Oak Ridge Leadership Computing Facility (OLCF), MIRA supercomputer at Argonne Leadership Computing Facilities (ALCF), Supercomputer at the National Research Center Kurchatov Institute , IT4 in Ostrava and others). Current approach utilizes modified PanDA pilot framework for job submission to the supercomputers batch queues and local data management, with light-weight MPI wrappers to run single threaded workloads in parallel on LCFs multi-core worker nodes. This implementation was tested with a variety of Monte-Carlo workloads on several supercomputing platforms for ALICE and ATLAS experiments and it is in full production for the ATLAS experiment since September 2015. We will present our current accomplishments with running PanDA WMS at supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facilities infrastructure for High Energy and Nuclear Physics as well as other data-intensive science applications, such as bioinformatics and astro-particle physics.

Data-intensive science↗
DOE OSTI2017

Mapping of exhaust gas recirculation and combustion-residual backflow in the intake ports of a heavy-duty diesel engine using a multiplexed multi-species absorption spectroscopy sensor [Mapping of EGR and combustion residual backflow in the intake ports of a heavy duty diesel engine using a multiplexed multi-species absorption spectroscopy sensor]

Here, the combustion-residual backflow into the intake ports of a commercial diesel engine (Cummins ISX series) was spatiotemporally mapped using a multiplexed multi-species absorption spectroscopy sensor system; the resulting cycle- and cylinder-resolved measurements are applicable for assessing cylinder charge uniformity, control strategies, and computational fluid dynamics tools. On-engine measurements were made using four compact (3/8 in Outside Diameter) stainless steel probes which enabled simultaneous multi-point measurements, required minimal engine hardware modification, and featured a novel tip design for measurement of gas flows parallel to the probe axis. Three sensor probes were used to perform simultaneous backflow measurements in intake runners corresponding to three of the six engine cylinders, and a fourth probe was installed in the intake manifold plenum for tracking dynamics introduced by an external exhaust gas recirculation mixer. Near-crank-angle resolved measurements (5 kHz, that is, 1.2 crank angle resolution at 1000 RPM) were performed during steady-state engine operation at various levels of external exhaust gas recirculation to measure the gas properties and penetration distance of the backflow into the intake runners on a cylinder- and cycle-basis. Validation of computational fluid dynamics model results is also presented to demonstrate the utility of such measurements in advancing engine research.

33 ADVANCED PROPULSION SYSTEMS↗
DOE OSTI2020

Analyzing the Performance Trade-Off in Implementing User-Level Threads

User-level threads have been widely adopted as a means of achieving lightweight concurrent execution without the costs of OS-level threads. Nevertheless, the costs of managing user-level threads represent a performance barrier that dictates how fine grained the concurrency exposed by an application can be without incurring significant overheads; this in turn may translate into insufficient parallelism to exploit highly parallel systems. This article is a deep dive into the fundamental costs in implementing user-level threads. We first identify that one of the highest sources of fork-join overheads stems from deviations, events that incur context switching during the execution of a thread and disrupt a run-to-completion execution. We then conduct an in-depth investigation of a wide spectrum of methods with respect to how they handle deviations while covering both parent- and child-first scheduling policies. Our methodology involves a comprehensive instruction- and cache-level analysis of all methods on several modern CPU architectures. Finally, the primary finding of our evaluation is that dynamic promotion methods that assume the absence of deviation and dynamically provide context-switching support offer the best trade-off between performance and capability when the likelihood of deviation is low.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2018

Center for Extended Magnetohydrodrynamic Modeling

This work used the NIMROD code to solve two tokamak regimes found on the DIII-D tokamak. The first regime studied is the Quiescent H-mode regime that has the advantage of enabling tokamak high performance without the deleterious effects of Edge Localized Modes (ELMs). ELMs are a possible impediment to the success of tokamaks as a burning plasma reactor and the QH mode regime provides a mechanism for avoiding it. The QH mode achieves this state due to the appearance of long wavelength instability that saturates instead of reaching an ELM state. NIMROD simulations were able to describe many of the features of a particular discharge, including the importance of shear flow in reaching the state, the role of the parallel heat flux to decorrelate the temperature and velocity fluctations, and in the correlation of density and velocity fluctations to provide the density pumpout. This work resulted in two direct publications and an invited talk. Progress towards these publications resulted in two more publications (verification and solving the Grad-Shafranov equation with scrape-off-layer currents). The second regime studied was the Giant Sawtooth regime whereby energetic particles affect the stability of the 1/1 mode. This work resulted in performance improvements of NIMROD's energetic particle algorithms, and improvements to the drift-kinetic equation infrastructure used by both continuum and energetic particle closure methods. The studies proved more computationally demanding than expected because as the 1/1 is stabilized, the near marginal conditions cause long simulations times. Publication of this work is in progress under other funding.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2017

Performance Analysis of OpenSHMEM Applications with TAU Commander

The TAU Performance System® (TAU) is a powerful and highly versatile profiling and tracing tool ecosystem for performance engineering of parallel programs. Developed over the last twenty years, TAU has evolved with each new generation of HPC systems and scales efficiently to hundreds of thousands of cores. TAU’s organic growth has resulted in a loosely coupled software toolbox such that novice users first encountering TAU’s complexity and vast array of features are often intimidated and easily frustrated. To lower the barrier to entry for novice TAU users, ParaTools and the US Department of Energy have developed “TAU Commander,” a performance engineering workflow manager that facilitates a systematic approach to performance engineering, guides users through common profiling and tracing workflows, and offers constructive feedback in case of error. This work compares TAU and TAU Commander workflows for common performance engineering tasks in OpenSHMEM applications and demonstrates workflows targeting two different SHMEM implementations, Intel Xeon “Haswell” and “Knights Landing” processors, direct and indirect measurement methods, callsite, profiles, and traces.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Model-parallel Fourier neural operators as learned surrogates for large-scale parametric PDEs

Fourier neural operators (FNOs) are a recently introduced neural network architecture for learning solution operators of partial differential equations (PDEs), which have been shown to perform significantly better than comparable deep learning approaches. Once trained, FNOs can achieve speed-ups of multiple orders of magnitude over conventional numerical PDE solvers. However, due to the high dimensionality of their input data and network weights, FNOs have so far only been applied to two-dimensional or small three-dimensional problems. To remove this limited problem-size barrier, we propose a model-parallel version of FNOs based on domain-decomposition of both the input data and network weights. Here, we demonstrate that our model-parallel FNO is able to predict time-varying PDE solutions of over 2.6 billion variables on Perlmutter using up to 512 A100 GPUs and show an example of training a distributed FNO on the Azure cloud for simulating multiphase CO 2 dynamics in the Earth’s subsurface.

58 GEOSCIENCES↗
DOE OSTI2026

Quality Control of Silicon Sensor Modules for Particle Detectors

The High-Luminosity Large Hadron Collider (HL-LHC) will produce a higher rate of particle collisions than the current Large Hadron Collider (LHC), requiring important upgrades to the Compact Muon Solenoid (CMS) to handle an increased amount of data. An important upgrade is the Phase-2 Outer Tracker Upgrade, which consists of 13,000 silicon sensor modules made of two parallel silicon sensors and readout electronics. These modules undergo careful quality control checks both during and after module assembly to ensure precise and reliable detector performance. This project focuses on precision testing for quality control of silicon sensor modules at Fermilab. Hands-on work includes visual inspection, current-voltage testing, module testing, and ultraviolet (UV) light exposure of modules showing abnormal current-voltage behavior. The ultraviolet exposure process improves the abnormal sensor readout data by placing the selected sensor side of the module directly under the UV light inside a controlled box. In addition to laboratory testing and ultraviolet experiments, I developed a Python-based data tool that connects to a module database and allows selected testing conditions and module information to be retrieved and displayed efficiently. These different testing procedures, experimental processes, and computational tools support the broader goal of identifying module issues and improving modules that will be used in the CMS Outer Tracker Phase-2 Upgrade.

Siddiqui, Hooriya [DuPage Coll.] (ORCID:0009000151↗
DOE OSTI2026

Quality Control of Silicon Sensor Modules for Particle Detectors

The High-Luminosity Large Hadron Collider (HL-LHC) will produce a higher rate of particle collisions than the current Large Hadron Collider (LHC), requiring significant upgrades to the Compact Muon Solenoid (CMS) to handle the increased amount of data. An important upgrade is the Phase-2 Outer Tracker Upgrade, which consists of 13,000 silicon sensor modules made of two parallel silicon sensors and readout electronics. These modules undergo careful quality control checks both during and after module assembly to ensure precise and reliable detector performance. This project focuses on precision testing for quality control of silicon sensor modules at Fermilab. Hands-on work includes visual inspection, current-voltage testing, module testing, and ultraviolet (UV) light exposure of modules showing abnormal current-voltage behavior. The ultraviolet exposure process improves the abnormal sensor readout data by placing the selected sensor side of the module directly under the UV light inside a controlled box. In addition to laboratory testing and ultraviolet experiments, I developed a Python-based data tool that connects to a module database and allows selected testing conditions and module information to be retrieved and displayed efficiently. These different testing procedures, experimental processes, and computational tools support the broader goal of identifying module issues and improving modules that will be used in the CMS Outer Tracker Phase-2 Upgrade.

Siddiqui, Hooriya [DuPage Coll.; Fermilab] (ORCID:↗
DOE OSTI2013

Parallel transport studies of high-Z impurities in the core of Alcator C-Mod plasmas

Measurements of poloidal variation, ñ{sub z}/ , in high-Z impurity density have been made using photodiode arrays sensitive to vacuum ultraviolet and soft x-ray emission in Alcator C-Mod plasmas. In/out asymmetries in the range of −0.2 <0.3 are observed for r/a<0.8, and accumulation on both the high-field side, n{sub z,cos}<0, and low-field side, n{sub z,cos}>0, of a flux surface is found to be well described by a combination of centrifugal, poloidal electric field, and ion-impurity friction effects. Up/down asymmetries, −0.05 <0.10, are observed over 0.5 0 corresponding to accumulation opposite the ion ∇B drift direction. Measurements of the up/down asymmetry of molybdenum are found to disagree with predictions from recent neoclassical theory in the trace limit, n{sub z}Z{sup 2}/n{sub i}≪1. Non-trace levels of impurities are expected to modify the main-ion poloidal flow and thus change friction-driven impurity density asymmetries and impurity poloidal rotation, v{sub θ,z}. Artificially modifying main-ion flow in parallel transport simulations is shown to impact both ñ{sub z}/ and v{sub θ,z}, but simultaneous agreement between measured and predicted up/down and in/out asymmetry as well as impurity poloidal rotation is not possible for these C-Mod data. This link between poloidal flow and poloidal impurity density variation outlines a more stringent test for parallel neoclassical transport theory than has previously been performed. Measurement and computational techniques specific to the study of poloidal impurity asymmetry physics are discussed as well.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗
DOE OSTI2024

GPU-enabled extreme-scale turbulence simulations: Fourier pseudo-spectral algorithms at the exascale using OpenMP offloading

Fourier pseudo-spectral methods for nonlinear partial differential equations are of wide interest in many areas of advanced computational science, including direct numerical simulation of three-dimensional (3-D) turbulence governed by the Navier-Stokes equations in fluid dynamics. This paper presents a new capability for simulating turbulence at a new record resolution up to 35 trillion grid points, on the world's first exascale computer, Frontier, comprising AMD MI250x GPUs with HPE's Slingshot interconnect and operated by the US Department of Energy's Oak Ridge Leadership Computing Facility (OLCF). Key programming strategies designed to take maximum advantage of the machine architecture involve performing almost all computations on the GPU which has the same memory capacity as the CPU, performing all-to-all communication among sets of parallel processes directly on the GPU, and targeting GPUs efficiently using OpenMP offloading for intensive number-crunching including 1-D Fast Fourier Transforms (FFT) performed using AMD ROCm library calls. With 99% of computing power on Frontier being on the GPU, leaving the CPU idle leads to a net performance gain via avoiding the overhead of data movement between host and device except when needed for some I/O purposes. Memory footprint including the size of communication buffers for MPI_ALLTOALL is managed carefully to maximize the largest problem size possible for a given node count. Detailed performance data including separate contributions from different categories of operations to the elapsed wall time per step are reported for five grid resolutions, from 2048 3 on a single node to 32768 3 on 4096 or 8192 nodes out of 9408 on the system. Both 1D and 2D domain decompositions which divide a 3D periodic domain into slabs and pencils respectively are implemented. The present code suite (labeled by the acronym GESTS, GPUs for Extreme Scale Turbulence Simulations) achieves a figure of merit (in grid points per second) exceeding goals set in the Center for Accelerated Application Readiness (CAAR) program for Frontier. The performance attained is highly favorable in both weak scaling and strong scaling, with notable departures only for 2048 3 where communication is entirely intra-node, and for 32768 3 , where a challenge due to small message sizes does arise. Communication performance is addressed further using a lightweight test code that performs all-to-all communication in a manner matching the full turbulence simulation code. Performance at large problem sizes is affected by both small message size due to high node counts as well as dragonfly network topology features on the machine, but is consistent with official expectations of sustained performance on Frontier. Overall, although not perfect, the scalability achieved at the extreme problem size of 32768 3 (and up to 8192 nodes — which corresponds to hardware rated at just under 1 exaflop/sec of theoretical peak computational performance) is arguably better than the scalability observed using prior state-of-the-art algorithms on Frontier's predecessor machine (Summit) at OLCF. New science results for the study of intermittency in turbulence enabled by this code and its extensions are to be reported separately in the near future.

3D fast Fourier transform↗
DOE OSTI2021

X-composer: enabling cross-environments in-situ workflows between HPC and cloud

As large-scale scientific simulations and big data analyses become more popular, it is increasingly more expensive to store huge amounts of raw simulation results to perform post-analysis. To minimize the expensive data I/O, "in-situ" analysis is a promising approach, where data analysis applications analyze the simulation generated data on the fly without storing it first. However, it is challenging to organize, transform, and transport data at scales between two semantically different ecosystems due to the distinct software and hardware difference. To tackle these challenges, we design and implement the X-Composer framework. X-Composer connects cross-ecosystem applications to form an "in-situ" scientific workflow, and provides a unified approach and recipe for supporting such hybrid in-situ workflows on distributed heterogeneous resources. X-Composer reorganizes simulation data as continuous data streams and feeds them seamlessly into the Cloud-based stream processing services to minimize I/O overheads. For evaluation, we use X-Composer to set up and execute a cross-ecosystem workflow, which consists of a parallel Computational Fluid Dynamics simulation running on HPC, and a distributed Dynamic Mode Decomposition analysis application running on Cloud. Our experimental results show that X-Composer can seamlessly couple HPC and Big Data jobs in their own native environments, achieve good scalability, and provide high-fidelity analytics for ongoing simulations in real-time.

Wang, Dali↗
DOE OSTI2013

Evaluating the performance of parallel subsurface simulators: An illustrative example with PFLOTRAN

To better inform the subsurface scientist on the expected performance of parallel simulators, this work investigates performance of the reactive multiphase flow and multicomponent biogeochemical transport code PFLOTRAN as it is applied to several realistic modeling scenarios run on the Jaguar supercomputer. After a brief introduction to the code's parallel layout and code design, PFLOTRAN's parallel performance (measured through strong and weak scalability analyses) is evaluated in the context of conceptual model layout, software and algorithmic design, and known hardware limitations. PFLOTRAN scales well (with regard to strong scaling) for three realistic problem scenarios: (1) in situ leaching of copper from a mineral ore deposit within a 5-spot flow regime, (2) transient flow and solute transport within a regional doublet, and (3) a real-world problem involving uranium surface complexation within a heterogeneous and extremely dynamic variably saturated flow field. Weak scalability is discussed in detail for the regional doublet problem, and several difficulties with its interpretation are noted.

54 ENVIRONMENTAL SCIENCES↗