Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “supercomputers”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 973 records · Page 54

A scalable nonlinear fluid–structure interaction solver based on a Schwarz preconditioner with isogeometric unstructured coarse spaces in 3D

Nonlinear fluid-structure interaction (FSI) problems on unstructured meshes in 3D appear many applications in science and engineering, such as vibration analysis of aircrafts and patient-specific diagnosis of cardiovascular diseases. In this work, we develop a highly scalable, parallel algorithmic and software framework for FSI problems consisting of a nonlinear fluid system and a nonlinear solid system, that are coupled monolithically. The FSI system is discretized by a stabilized finite element method in space and a fully implicit backward difference scheme in time. To solve the large, sparse system of nonlinear algebraic equations at each time step, we propose an inexact Newton-Krylov method together with a multilevel, smoothed Schwarz preconditioner with isogeometric coarse meshes generated by a geometry preserving coarsening algorithm. Here ''geometry'' includes the boundary of the computational domain and the wet interface between the fluid and the solid. We show numerically that the proposed algorithm and implementation are highly scalable in terms of the number of linear and nonlinear iterations and the total compute time on a supercomputer with more than 10,000 processor cores for several problems with hundreds of millions of unknowns.

97 MATHEMATICS AND COMPUTING↗

Modeling of wave processes in blocky media with porous and fluid-saturated interlayers

The wave processes in blocky media are analyzed by applying different mathematical models, wherein the elastic blocks interact with each other via pliant interlayers with the complex mechanical properties. Four versions of constitutive equations are considered. In the first version, an elastic interaction between the blocks is simulated within the framework of linear elasticity theory, and the model of elastic–plastic interlayers is constructed to take into account the appearance of irreversible deformation of interlayers at short time intervals. In the second one, the effects of viscoelastic shear in the interblock interlayers are taken into the consideration using the Poynting–Thomson rheological scheme. In the third option, the model of an elastic porous material is used in the interlayers, where the pores collapse if an abrupt compressive stress is applied. In the fourth case, the model of a fluid-saturated material with open pores is examined based on Biot's equations. The collapse of pores is modeled by the generalized rheological approach, wherein the mechanical properties of a material are simulated using four rheological elements. Three of them are the traditional elastic, viscous and plastic elements, the fourth element is the so-called rigid contact, which is used to describe the behavior of materials with the different resistance to tension and compression. It was shown that the thermodynamically consistent model is provided, which means that the energy balance equation is fulfilled for an entire blocky structure, where the kinetic and potential energy of the system is the sum of the kinetic and potential energies of the blocks and interlayers. Under numerical implementation of the interlayers models, the dissipationless finite difference Ivanov's method was used. The splitting method by spatial variables in the combination with the Godunov gap decay scheme was applied in the blocks. As a result, robust and stable computational algorithms are built and tested. Using MPI technology, the parallel software was designed for the modeling of wave processes in 2D setting. The numerical results are presented, discussed and future studies are outlined. - Highlights: • We model the propagation of waves in a blocky medium with compliant interlayers. • Models of porous and fluid saturated interlayers are proposed using Biot's approach. • A parallel computational algorithm is developed for supercomputers of cluster type. • Problems of instant rotation of central and boundary blocks are solved numerically. • Wave motions are compared in the cases of elastic and fluid-saturated interlayers.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Stabilized bases for high-order, interpolation semi-Lagrangian, element-based tracer transport

In a computational fluid model of the atmosphere, the advective transport of trace species, or tracers, can be computationally expensive. For efficiency, models often use semi-Lagrangian advection methods. High-order interpolation semi-Lagrangian (ISL) methods, in particular, can be extremely efficient, if the problem of property preservation specific to them can be addressed. Atmosphere models often use geometrically and logically nonuniform grids for efficiency and, as a result, element-based discretizations. Such grids and discretizations make stability a particular problem for ISL methods. Generally, high-order, element-based ISL methods that use the natural polynomial interpolant associated with a nodal finite-element discretization are unstable. Here, we derive new bases having order of accuracy up to nine, with positive nodal weights, that stabilize the element-based ISL method. We use these bases to construct the linear advection operator in the property-preserving Interpolation Semi-Lagrangian Element-based Transport (Islet) method. Then we discuss key software implementation details. Finally, we show performance results for the Energy Exascale Earth System Model's atmosphere dynamical core, comparing the original and new transport methods. These simulations used up to 27,600 Graphical Processing Units (GPU) on the Oak Ridge Leadership Computing Facility's Summit supercomputer.

97 MATHEMATICS AND COMPUTING↗

A prospect for computing in porous materials research: Very large fluid flow simulations

Properties of porous materials, abundant both in nature and industry, have broad influences on societies via, e.g. oil recovery, erosion, and propagation of pollutants. The internal structure of many porous materials involves multiple scales which hinders research on the relation between structure and transport properties: typically laboratory experiments cannot distinguish contributions from individual scales while computer simulations cannot capture multiple scales due to limited capabilities. Thus the question arises how large domain sizes can in fact be simulated with modern computers. This question is here addressed using a realistic test case; it is demonstrated that current computing capabilities allow the direct pore-scale simulation of fluid flow in porous materials using system sizes far beyond what has been previously reported. The achieved system sizes allow the closing of some particular scale gaps in, e.g. soil and petroleum rock research. Specifically, a full steady-state fluid flow simulation in a porous material, represented with an unprecedented resolution for the given sample size, is reported: the simulation is executed on a CPU-based supercomputer and the 3D geometry involves 16,3843 lattice cells (around 590 billion of them are pore sites). Using half of this sample in a benchmark simulation on a GPU-based system, a sustained computational performance of 1.77 PFLOPS is observed. These advances expose new opportunities in porous materials research. The implementation techniques here utilized are standard except for the tailored high-performance data layouts as well as the indirect addressing scheme with a low memory overhead and the truly asynchronous data communication scheme in the case of CPU and GPU code versions, respectively.

Computer Science↗

Translational research in the MPICH project

The MPICH project is an example of translational research in computer science before that term was well known or even coined. The project began in 1992 as an effort to develop a portable, high-performance implementation of the emerging Message-Passing Interface (MPI) Standard. It has enabled the widespread adoption of MPI as a way to write scalable parallel applications on systems of all sizes including upcoming exascale supercomputers. In this paper, we describe how the translational research process was used in MPICH, how that led to its success, the challenges encountered and lessons learned, and how the process could be applied to other similar projects.

97 MATHEMATICS AND COMPUTING↗

Optimizing the hypre solver for manycore and GPU architectures

The solution of large-scale combustion problems with codes such as Uintah on modern computer architectures requires the use of multithreading and GPUs to achieve performance. Uintah uses a low-Mach number approximation that requires iteratively solving a large system of linear equations. The Hypre iterative solver has solved such systems in a scalable way for Uintah, but the use of OpenMP with Hypre leads to at least slowdown due to OpenMP overheads. The proposed solution uses the MPI Endpoints within Hypre, where each team of threads acts as a different MPI rank. This approach minimizes OpenMP synchronization overhead and performs as fast or (up to 1.44) faster than Hypre's MPI-only version, and allows the rest of Uintah to be optimized using OpenMP. The profiling of the GPU version of Hypre shows the bottleneck to be the launch overhead of thousands of micro-kernels. The GPU performance was improved by fusing these micro-kernels and was further optimized by using Cuda-aware MPI, resulting in an overall speedup of 1.16—1.44 compared to the baseline GPU implementation. The above optimization strategies were published in the International Conference on Computational Science 2020 [1]. This work extends the previously published research by carrying out the second phase of communication-centered optimizations in Hypre to improve its scalability on large-scale supercomputers. Additionally, this includes an efficient non-blocking inter-thread communication scheme, communication-reducing patch assignment, and expression of logical communication parallelism to a new version of the MPICH library that utilizes the underlying network parallelism [2]. The above optimizations avoid communication bottlenecks previously observed during strong scaling and improve performance by up to 2 on 256 nodes of Intel Knight's Landing processor.

97 MATHEMATICS AND COMPUTING↗

Optimal checkpointing for adjoint multistage time-stepping schemes

Here, we consider checkpointing strategies that minimize the number of recomputations needed when performing discrete adjoint computations using multistage time-stepping schemes that require computing several substeps within one complete time step. Specifically, we propose two algorithms that can generate optimal checkpoint-ing schedules under weak assumptions. The first is an extension of the seminal Revolve algorithm adapted to multistage schemes. The second algorithm, named CAMS, is developed based on dynamic programming, and it requires the least number of recomputations when compared with other algorithms. The CAMS algorithm is made publicly available in a library with bindings to C and Python. Numerical results show that the proposed algorithms can deliver up to two times the speedup compared with that of classical Revolve. Moreover, we discuss the utilization of the CAMS library in mature scientific computing libraries and demonstrate the ease of using it in an adjoint workflow. The proposed algorithms have been adopted by the PETSc TSAdjoint library. Their performance has been demonstrated with a large-scale PDE-constrained optimization problem on a leadership-class supercomputer. This work is a significant extension of the authors' conference paper.

97 MATHEMATICS AND COMPUTING↗

An ultrahigh-resolution E3SM land model simulation framework and its first application to the Seward Peninsula in Alaska

The availability of supercomputers and state-of-science datasets has made it possible to conduct large-scale land simulations at an ultrahigh-resolution. This study reported a computational framework for land surface simulation using the E3SM land model (ELM) at an unprecedented resolution (1 km x 1 km gridcell). The ultrahigh-resolution ELM (uELM) simulation framework includes three parts: (1) high-resolution atmospheric forcing and surface properties dataset generation, (2) massive gridcell-based simulation, and (3) large-scale simulation results analysis. Additionally, we implemented the uELM simulation framework and completed the first 1 km x 1 km terrestrial ecosystem simulation (from 1850 to 2014) over the Seward Peninsula in Alaska (78,000 km 2 ). The experiment contained two phases: a spin-up simulation and a transient simulation, and required five weeks of calculations using 320 cores in a 44-node Linux HPC computer. It created approximately 1.3 TB of data from the transient simulation alone (1850 - present). We selected sample results (monthly and daily simulation outputs) to illustrate the temporal and spatial variations of several variables in high-latitude Arctic ecosystems’ water, energy, and carbon cycles. At last, we summarized the lessons learned and proposed new developments for full-scale uELM simulations over the entire North American continent (approximately 22,000,000 km 2 ).

54 ENVIRONMENTAL SCIENCES↗

An uncertainty visualization framework for large-scale cardiovascular flow simulations: A case study on aortic stenosis

We present a generalizable uncertainty quantification (UQ) and visualization framework for lattice Boltzmann method simulations of high Reynolds number vascular flows, demonstrated on a patient-specific stenosed aorta. The framework combines EasyVVUQ for parameter sampling with large-eddy simulation turbulence modeling in HemeLB, and executes ensembles on the Frontier exascale supercomputer. Spatially resolved metrics, including entropy and isosurface-crossing probability, are used to map uncertainty in pressure and wall shear stress fields directly onto vascular geometries. Two sources of model variability are examined: inlet peak velocity and the Smagorinsky constant. Inlet velocity variation produces high uncertainty downstream of the stenosis where turbulence develops, while upstream regions remain stable. Smagorinsky constant variation has little effect on the large-scale pressure field but increases WSS uncertainty in localized high-shear regions. In both cases, the stenotic throat manifests low entropy, indicative of robust identification of elevated WSS. By linking quantitative UQ measures to three-dimensional anatomy, the framework improves interpretability over conventional 1D UQ plots and supports clinically relevant decision-making, with broad applicability to vascular flow problems requiring both accuracy and spatial insight.

Hemodynamics↗

Reconstructing householder vectors from Tall-Skinny QR

The Tall-Skinny QR (TSQR) algorithm is more communication efficient than the standard Householder algorithm for QR decomposition of matrices with many more rows than columns. However, TSQR produces a different representation of the orthogonal factor and therefore requires more software development to support the new representation. Further, implicitly applying the orthogonal factor to the trailing matrix in the context of factoring a square matrix is more complicated and costly than with the Householder representation. We show how to perform TSQR and then reconstruct the Householder vector representation with the same asymptotic communication efficiency and little extra computational cost. We demonstrate the high performance and numerical stability of this algorithm both theoretically and empirically. The new Householder reconstruction algorithm allows us to design more efficient parallel QR algorithms, with significantly lower latency cost compared to Householder QR and lower bandwidth and latency costs compared with Communication-Avoiding QR (CAQR) algorithm. Experiments on supercomputers demonstrate the benefits of the communication cost improvements: in particular, our experiments show substantial improvements over tuned library implementations for tall-and-skinny matrices. Furthermore, we also provide algorithmic improvements to the Householder QR and CAQR algorithms, and we investigate several alternatives to the Householder reconstruction algorithm that sacrifice guarantees on numerical stability in some cases in order to obtain higher performance.

97 MATHEMATICS AND COMPUTING↗

Advances in Patch-Based Adaptive Mesh Refinement Scalability

Patch-based structured adaptive mesh refinement (SAMR) is widely used for high-resolution simu- lations. Combined with modern supercomputers, it could provide simulations of unprecedented size and resolution. A persistent challenge for this com- bination has been managing dynamically adaptive meshes on more and more MPI tasks. The dis- tributed mesh management scheme in SAMRAI has made some progress SAMR scalability, but early al- gorithms still had trouble scaling past the regime of 105 MPI tasks. This work provides two critical SAMR regridding algorithms, which are integrated into that scheme to ensure efficiency of the whole. The clustering algorithm is an extension of the tile- clustering approach, making it more flexible and efficient in both clustering and parallelism. The partitioner is a new algorithm designed to prevent the network congestion experienced by its prede- cessor. We evaluated performance using weak- and strong-scaling benchmarks designed to be difficult for dynamic adaptivity. Results show good scaling on up to 1.5M cores and 2M MPI tasks. Detailed timing diagnostics suggest scaling would continue well past that.

97 MATHEMATICS AND COMPUTING↗

Massively parallel first-principles simulation of electron dynamics in materials

Here we present a highly scalable, parallel implementation of first-principles electron dynamics coupled with molecular dynamics (MD). By using optimized kernels, network topology aware communication, and by fully distributing all terms in the time-dependent Kohn–Sham equation, we demonstrate unprecedented time to solution for disordered aluminum systems of 2000 atoms (22,000 electrons) and 5400 atoms (59,400 electrons), with wall clock time as low as 7.5 s per MD time step. Despite a significant amount of non-local communication required in every iteration, we achieved excellent strong scaling and sustained performance on the Sequoia Blue Gene/Q supercomputer at LLNL. We obtained up to 59% of the theoretical sustained peak performance on 16,384 nodes and performance of 8.75 Petaflop/s (43% of theoretical peak) on the full 98,304 node machine (1,572,864 cores). Lastly, scalable explicit electron dynamics allows for the study of phenomena beyond the reach of standard first-principles MD, in particular, materials subject to strong or rapid perturbations, such as pulsed electromagnetic radiation, particle irradiation, or strong electric currents.

36 MATERIALS SCIENCE↗

An automatic performance model-based scheduling tool for coupled climate system models

The prediction of the climate system is highly depended on the efficient integration of observations and simulations of the Earth. This is regarded as a canonical example of the cyber-physical system. The climate system model, the simulation engine in this cyber-physical system, is one of most challenging applications in scientific computing. It utilizes the multi-physics simulation that couples multiple components, conducts decadal to millennium simulations, and has long been an important application on supercomputers. However, current climate system models suffer from the inefficient task scheduling methods resulting in an intolerable simulation time. Take the Community Earth System Model (CESM), the most widely used climate system model, as an example, one major reason that CESM suffers from bad performances is the huge overhead to rationally distribute processes among the coupled heterogeneous components. According to the report of NCAR, every percent improvement in CESM performance frees up to the equivalent of $250,000 in computing resources in their scientific experiments. To address such challenge, our paper firstly constructs a lightweight and accurate performance model for effectively capturing and predicting the heterogeneous time-to-solution performance of end-to-end CESM components with a given simulation configuration. Then, based on the performance model, we further propose an efficient scheduling strategy based on rectangular packing method to determine the best process layout among different components, and the process numbers assigned to each component. Our evaluations show that we can achieve 58% average run time reductions on CESM comparing to the widely used sequential process layout for a scale of 144 to 480 cores on typical CPU clusters. And we can save 4 million CPU hours when we conduct one standard scientific experiment (a 2870-year simulation), which equals to save $40,089 with a charge of $0.01 per CPU hour. Meanwhile, 26% extra performance improvements could also be gained in our methods comparing to the heuristic branch and bound algorithm with the guidance of the known curve-fitting performance model.

58 GEOSCIENCES↗

Parallel algorithms for finding connected components using linear algebra

Finding connected components is one of the most widely used operations on a graph. Optimal serial algorithms for the problem have been known for half a century, and many competing parallel algorithms have been proposed over the last several decades under various different models of parallel computation. This paper presents a class of parallel connected-component algorithms designed using linear-algebraic primitives. These algorithms are based on a PRAM algorithm by Shiloach and Vishkin and can be designed using standard GraphBLAS operations. Here, we demonstrate two algorithms of this class, one named LACC for Linear Algebraic Connected Components, and the other named FastSV which can be regarded as LACC’s simplification. With the support of the highly-scalable Combinatorial BLAS library, LACC and FastSV outperform the previous state-of-the-art algorithm by a factor of up to 12x for small to medium scale graphs. For large graphs with more than 50B edges, LACC and FastSV scale to 4K nodes (262K cores) of a Cray XC40 supercomputer and outperform previous algorithms by a significant margin. This remarkable performance is accomplished by (1) exploiting sparsity that was not present in the original PRAM algorithm formulation, (2) using high-performance primitives of Combinatorial BLAS, and (3) identifying hot spots and optimizing them away by exploiting algorithmic insights.

97 MATHEMATICS AND COMPUTING↗

An empirical study of I/O separation for burst buffers in HPC systems

To meet the exascale I/O requirements for the High-Performance Computing (HPC), a new I/O subsystem, Burst Buffer, based on solid state drives (SSD), has been developed. However, the diverse HPC workloads and the bursty I/O pattern cause severe data fragmentation that requires costly garbage collection (GC) and increases the number of bytes written to the SSD. To address this data fragmentation challenge, a new multi-stream feature has been developed for SSDs. In this work, we develop an I/O Separation scheme called BIOS to leverage this multi-stream feature to group the I/O streams based on the user IDs. We propose a stream-aware scheduling policy based on burst buffer pools in the workload manager, and integrate the BIOS with the workload manager to optimize the I/O separation scheme in burst buffer. We evaluate the proposed framework with a burst buffer I/O traces from Cori Supercomputer including a diverse set of applications. Experimental results show that the BIOS could improve the performance by 1.44x on average and reduce the Write Amplification Factor (WAF) by up to 1.20x. Finally, these demonstrate the potential benefits of the I/O separation scheme for solid state storage systems.

97 MATHEMATICS AND COMPUTING↗

Study of interconnect errors, network congestion, and applications characteristics for throttle prediction on a large scale HPC system

Today’s High Performance Computing (HPC) systems contain thousand of nodes which work together to provide performance in the order of petaflops. The performance of these systems depends on various components like processors, memory, and interconnect. Among all, interconnect plays a major role as it glues together all the hardware components in an HPC system. A slow interconnect can impact a scientific application running on multiple processes severely as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks a study that explores different interconnect errors, congestion events and applications characteristics on a large-scale HPC system. In our previous work, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors, and congestion events. In this work, we first show how congestion events can impact application performance. We then investigate application characteristics interaction with interconnect errors and network congestion to predict applications encountering congestion with more than 90% accuracy.

97 MATHEMATICS AND COMPUTING↗