Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “parallel, scaling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 181 records · Page 10

Early experiences on the OLCF Frontier system with AthenaPK and Parthenon–Hydro

The Oak Ridge Leadership Computing Facility (OLCF) has been preparing the nation's first exascale system, Frontier, for production and end users. Frontier is based on HPE Cray's new EX architecture and Slingshot interconnect and features 74 cabinets of optimized 3rd Gen AMD EPYC CPUs for HPC and AI and AMD Instinct 250X accelerators. As a part of this preparation, “real-world” user codes have been selected to help assess the functionality, performance, and usability of the system. This article describes early experiences using the system in collaboration with the Hamburg Observatory for two selected codes, which have since been adopted in the OLCF test harness. Experiences discussed include efforts to resolve performance variability and per-cycle slowdowns. Results are shown for a performance portable astrophysical magnetohydronamics code, AthenaPK, and a mini-application stressing the core functionality of a performance portable block-structured adaptive mesh refinement framework, Parthenon-Hydro. Here, these results show good scaling characteristics to the full system. At the largest scale, the Parthenon-Hydro miniapp reaches a total of $1.7$ $\times$ $10^{13}$ zone-cycles/s on 9216 nodes (73,728 logical GPUs) at ≈92% weak scaling parallel efficiency (starting from a single node using a second-order, finite-volume method).

97 MATHEMATICS AND COMPUTING↗

Accuracy of finite-difference harmonic frequencies in density functional theory

Analytic Hessians are often viewed as essential for the calculation of accurate harmonic frequencies, but the implementation of analytic second derivatives is nontrivial and solution of the requisite coupled-perturbed equations engenders a sizable memory footprint for large systems, given that these equations are not required for energy and gradient calculations in density functional theory. Here, we benchmark the alternative approach to harmonic frequencies based on finite differences of analytic first derivatives, a procedure that is amenable to large-scale parallelization. Not only for absolute frequencies but also for isotopic and conformer-dependent frequency shifts in flexible molecules, we find that the finite-difference approach exhibits mean errors < 0.1 cm -1 as compared to results based on an analytic Hessian. For very small frequencies corresponding to nonbonded vibrations in noncovalent complexes (for which the harmonic approximation is questionable anyway), the finite-difference error can be larger, but even in these cases the errors can be reduced below 0.1 cm -1 by judicious choice of the displacement step size and a higher-order finite-difference approach. Finally, the surprising accuracy and robustness of the finite-difference results suggests that availability of the analytic Hessian is not so important in today's era of commodity processors that are readily available in large numbers.

74 ATOMIC AND MOLECULAR PHYSICS↗

Towards a Standard Process Management Infrastructure for Workflows Using Python

Orchestrating the execution of ensembles of processes lies at the core of scientific workflow engines on large scale parallel platforms. This is usually handled using platform-specific command line tools, with limited process management control and potential strain on system resources. The PMIx standard provides a uniform interface to system resources. The low level C implementation of PMIx has hampered its use in workflow engines, leading to the development of Python binding that has yet to gain traction. In this paper, we present our work to harden the PMIx Python client, demonstrating its usability using a prototype Python driver to orchestrate the execution of an ensemble of processes. We present experimental results using the prototype on the Summit supercomputer at Oak Ridge National Laboratory. This work lays the foundation for wider adoption of PMIx for workflow engines, and encourages wider support of more PMIx functionality in vendor provided system software stacks.

Elwasif, Wael↗

Hierarchical computation in the SPMD programming model

Large-scale parallel machines are programmed mainly with the single program, multiple data (SPMD) model of parallelism. While this model has advantages of scalability and simplicity, it does not fit well with divide-and-conquer parallelism or hierarchical machines that mix shared and distributed memory. In this paper, we define the recursive single program, multiple data model (RSPMD) that extends SPMD with a hierarchical team mechanism to support hierarchical algorithms and machines. We implement this model in the Titanium language and describe how to eliminate a class of deadlocks by ensuring alignment of collective operations. We present application case studies evaluating the RSPMD model, showing that it enables divide-and-conquer algorithms such as sorting to be elegantly expressed and that team collective operations increase performance of conjugate gradient by up to a factor of two. The model also facilitates optimizations for hierarchical machines, improving scalability of particle in cell by 8x and performance of sorting and a stencil code by up to 40% and 14%, respectively.

Kamil, A↗

Experience and Practice of Batch Scheduling on Leadership Supercomputers at Argonne

The mission of the DOE Argonne Leadership Computing Facility (ALCF) is to accelerate major scientific discoveries and engineering breakthroughs for humanity by designing and providing world-leading computing facilities in partnership with the computational science community. The ALCF operates supercomputers that are generally amongst the Top 5 fastest machines in the world. Specifically, ALCF is looking for the science that is either too big to run anywhere else, or it would take so long as to be impractical (i.e., “capability jobs”). At ALCF, batch scheduling plays a critical role for achieving a set of site goals within a set of constraints. While system utilization is an important goal at ALCF, its largest mission constraint is to enable extreme scale parallel jobs to take precedence. In this paper, we will describe the specific scheduling goals and constraints, analyze the workload traces collected in 2013–2017 from the 48-rack petascale supercomputer Mira, and discuss the upcoming scheduling challenges at ALCF.

Allcock, William↗

Towards reversible basic linear algebra subprograms: A performance study

Problems such as fault tolerance and scalable synchronization can be efficiently solved using reversibility of applications. Making applications reversible by relying on computation rather than on memory is ideal for large scale parallel computing, especially for the next generation of supercomputers in which memory is expensive in terms of latency, energy, and price. In this direction, a case study is presented here in reversing a computational core, namely, Basic Linear Algebra Subprograms, which is widely used in scientific applications. A new Reversible BLAS (RBLAS) library interface has been designed, and a prototype has been implemented with two modes: (1) a memory-mode in which reversibility is obtained by checkpointing to memory in forward and restoring from memory in reverse, and (2) a computational-mode in which nothing is saved in the forward, but restoration is done entirely via inverse computation in reverse. The article is focused on detailed performance benchmarking to evaluate the runtime dynamics and performance effects, comparing reversible computation with checkpointing on both traditional CPU platforms and recent GPU accelerator platforms. For BLAS Level-1 subprograms, data indicates over an order of magnitude better speed of reversible computation compared to checkpointing. For BLAS Level-2 and Level-3, a more complex tradeoff is observed between reversible computation and checkpointing, depending on computational and memory complexities of the subprograms.

97 MATHEMATICS AND COMPUTING↗

The Peridigm Meshfree Peridynamics Code

Abstract Peridigm is a meshfree peridynamics code written in C++ for use on large-scale parallel computers. It was originally developed at Sandia National Laboratories and is currently managed as an open-source, community driven software project. Its primary features include bond-based, state-based, and non-ordinary state-based constitutive models, bond failure laws, contact, and support for explicit and implicit time integration. To date, Peridigm has been used primarily by methods developers focused on solid mechanics and material failure. Peridigm utilizes foundational software components from Sandia’s Trilinos project and was designed for extensibility. This paper provides an overview of the solution methods implemented in Peridigm , a discussion of its software infrastructure, and demonstrates the use of Peridigm for the solution of several example problems.

97 MATHEMATICS AND COMPUTING↗

Logical design of anti-prion agents using NAGARA

To accelerate the logical drug design procedure, we created the program “NAGARA,” a plugin for PyMOL, and applied it to the discovery of small compounds called medical chaperones (MCs) that stabilize the cellular form of a prion protein (PrP{sup C}). In NAGARA, we constructed a single platform to unify the docking simulation (DS), free energy calculation by molecular dynamics (MD) simulation, and interfragment interaction energy (IFIE) calculation by quantum chemistry (QC) calculation. NAGARA also enables large-scale parallel computing via a convenient graphical user interface. Here, we demonstrated its performance and its broad applicability from drug discovery to lead optimization with full compatibility with various experimental methods including Western blotting (WB) analysis, surface plasmon resonance (SPR), and nuclear magnetic resonance (NMR) measurements. Combining DS and WB, we discovered anti-prion activities for two compounds and tegobuvir (TGV), a non-nucleoside non-structural protein NS5B polymerase inhibitor showing activity against hepatitis C virus genotype 1. Binding profiles predicted by MD and QC are consistent with those obtained by SPR and NMR. Free energy analyses showed that these compounds stabilize the PrP{sup C} conformation by decreasing the conformational fluctuation of the PrP{sup C}. Because TGV has been already approved as a medicine, its extension to prion diseases is straightforward. Finally, we evaluated the affinities of the fragmented regions of TGV using QC and found a clue for its further optimization. By repeating WB, MD, and QC recursively, we were able to obtain the optimum lead structure. - Highlights: • NAGARA integrates docking simulation, molecular dynamics, and quantum chemistry. • We found many compounds, e.g., tegobuvir (TGV), that exhibit anti-prion activities. • We obtained insights into the action mechanism of TGV as a medical chaperone. • Using QC, we obtained useful information for optimization of the lead compound, TGV. • NAGARA is a convenient platform for drug discovery and lead optimization.

60 APPLIED LIFE SCIENCES↗

Performance of fully-coupled algebraic multigrid preconditioners for large-scale VMS resistive MHD

Here, we explore the current performance and scaling of a fully-implicit stabilized unstructured finite element (FE) variational multiscale (VMS) capability for large-scale simulations of 3D incompressible resistive magnetohydrodynamics (MHD). The large-scale linear systems that are generated by a Newton nonlinear solver approach are iteratively solved by preconditioned Krylov subspace methods. The efficiency of this approach is critically dependent on the scalability and performance of the algebraic multigrid preconditioner. Our study considers the performance of the numerical methods as recently implemented in the second-generation Trilinos implementation that is 64-bit compliant and is not limited by the 32-bit global identifiers of the original Epetra-based Trilinos. The study presents representative results for a Poisson problem on 1.6 million cores of an IBM Blue Gene/Q platform to demonstrate very large-scale parallel execution. Additionally, results for a more challenging steady-state MHD generator and a transient solution of a benchmark MHD turbulence calculation for the full resistive MHD system are also presented. These results are obtained on up to 131,000 cores of a Cray XC40 and one million cores of a BG/Q system.

97 MATHEMATICS AND COMPUTING↗

A sensitivity analysis of twinning crystal plasticity finite element model using single crystal and poly crystal Zircaloy

The popularity of crystal plasticity finite element method (CPFEM) models is increasing due to their ability to predict the mechanical response of crystalline materials such as metals and metal alloys more accurately than traditional continuum mechanics models. This is since the crystal plasticity models consider the effect of atomic structure, microstructural morphology, and properties of individual grains. These CPFEM models use a large number of material parameters in order to capture the mesoscale physics which comes with the downside of the tedious calibration process. In this paper, a CPFEM code was developed to include the twinning induced grain reorientation and subsequent crystallographic slip for HPC material. The developed code is incorporated in a large-scale, parallelized nonlinear solver WARP3D. Further, a sensitivity analysis with respect to 22 material parameters was then conducted using single crystal and polycrystal representative volume element (RVE) of Zircaloy material. Loading was applied along five different crystallographic orientations for single crystal RVE and along three directions namely, rolling (RD), transverse (TD), and normal (ND) direction for polycrystal RVE. Results obtained from the sensitivity analysis were used for the calibration of material parameters for Zircaloy. Finally, developed code along with calibrated material parameters was used to investigate the effect of the hydride phase formation in Zircaloy which is a typical case observed for nuclear applications. It was found that the volume fraction of the hydride phase has a significant impact on the mechanical properties of Zircaloy.

36 MATERIALS SCIENCE↗

2011 Status of the Automatic Alignment System for the National Ignition Facility

Automated alignment for the National Ignition Facility (NIF) is accomplished using a large-scale parallel control system that directs 192 laser beams along the 300-m optical path. The beams are then focused down to a 50-micron spot in the middle of the target chamber. The entire process is completed in less than 50 minutes. The alignment system commands 9,000 stepping motors for highly accurate adjustment of mirrors and other optics. 41 control loops per beamline perform parallel processing services running on a LINUX cluster to analyze high-resolution images of the beams and their references. This paper describes the status the NIF automatic alignment system and the challenges encountered as NIF development has transitioned from building the laser, to becoming a research project supporting a 24 hour, 7 day laser facility. NIF is now a continuously operated system where performance monitoring is increasingly more critical for operation, maintenance, and commissioning tasks. Equipment wear and the effects of high energy neutrons from fusion experiments are issues which alter alignment efficiency and accuracy. New sensors needing automatic alignment assistance are common. System modifications to improve efficiency and accuracy are prevalent. Handling these evolving alignment and maintenance needs while minimizing the impact on NIF experiment schedule is expected to be an on-going challenge for the planned 30 year operational life of NIF.

42 ENGINEERING↗

A domain decomposition method for pseudo-spectral electromagnetic simulations of plasmas

Pseudo-spectral electromagnetic solvers (i.e. representing the fields in Fourier space) have extraordinary precision. In particular, Haber et al. presented in 1973 a pseudo-spectral solver that integrates analytically the solution over a finite time step, under the usual assumption that the source is constant over that time step. Yet, pseudo-spectral solvers have not been widely used, due in part to the difficulty for efficient parallelization owing to global communications associated with global FFTs on the entire computational domains. A method for the parallelization of electromagnetic pseudo-spectral solvers is proposed and tested on single electromagnetic pulses, and on Particle-In-Cell simulations of the wakefield formation in a laser plasma accelerator. The method takes advantage of the properties of the Discrete Fourier Transform, the linearity of Maxwell’s equations and the finite speed of light for limiting the communications of data within guard regions between neighboring computational domains. Although this requires a small approximation, test results show that no significant error is made on the test cases that have been presented. The proposed method opens the way to solvers combining the favorable parallel scaling of standard finite-difference methods with the accuracy advantages of pseudo-spectral methods.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

A discontinuous Galerkin conservative level set scheme for interface capturing in multiphase flows

The accurate conservative level set (ACLS) method of Desjardins et al. [O. Desjardins, V. Moureau, H. Pitsch, An accurate conservative level set/ghost fluid method for simulating turbulent atomization, J. Comput. Phys. 227 (18) (2008) 8395–8416] is extended by using a discontinuous Galerkin (DG) discretization. DG allows for the scheme to have an arbitrarily high order of accuracy with the smallest possible computational stencil resulting in an accurate method with good parallel scaling. This work includes a DG implementation of the level set transport equation, which moves the level set with the flow field velocity, and a DG implementation of the reinitialization equation, which is used to maintain the shape of the level set profile to promote good mass conservation. A near second order converging interface curvature is obtained by following a height function methodology (common amongst volume of fluid schemes) in the context of the conservative level set. Various numerical experiments are conducted to test the properties of the method and show excellent results, even on coarse meshes. The tests include Zalesak’s disk, two-dimensional deformation of a circle, time evolution of a standing wave, and a study of the Kelvin–Helmholtz instability. Finally, this novel methodology is employed to simulate the break-up of a turbulent liquid jet.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Anderson acceleration of the Jacobi iterative method: An efficient alternative to Krylov methods for large, sparse linear systems

We employ Anderson extrapolation to accelerate the classical Jacobi iterative method for large, sparse linear systems. Specifically, we utilize extrapolation at periodic intervals within the Jacobi iteration to develop the Alternating Anderson–Jacobi (AAJ) method. We verify the accuracy and efficacy of AAJ in a range of test cases, including nonsymmetric systems of equations. We demonstrate that AAJ possesses a favorable scaling with system size that is accompanied by a small prefactor, even in the absence of a preconditioner. In particular, we show that AAJ is able to accelerate the classical Jacobi iteration by over four orders of magnitude, with speed-ups that increase as the system gets larger. Moreover, we find that AAJ significantly outperforms the Generalized Minimal Residual (GMRES) method in the range of problems considered here, with the relative performance again improving with size of the system. As a result, the proposed method represents a simple yet efficient technique that is particularly attractive for large-scale parallel solutions of linear systems of equations.

97 MATHEMATICS AND COMPUTING↗

ColDICE: A parallel Vlasov–Poisson solver using moving adaptive simplicial tessellation

Resolving numerically Vlasov–Poisson equations for initially cold systems can be reduced to following the evolution of a three-dimensional sheet evolving in six-dimensional phase-space. We describe a public parallel numerical algorithm consisting in representing the phase-space sheet with a conforming, self-adaptive simplicial tessellation of which the vertices follow the Lagrangian equations of motion. The algorithm is implemented both in six- and four-dimensional phase-space. Refinement of the tessellation mesh is performed using the bisection method and a local representation of the phase-space sheet at second order relying on additional tracers created when needed at runtime. In order to preserve in the best way the Hamiltonian nature of the system, refinement is anisotropic and constrained by measurements of local Poincaré invariants. Resolution of Poisson equation is performed using the fast Fourier method on a regular rectangular grid, similarly to particle in cells codes. To compute the density projected onto this grid, the intersection of the tessellation and the grid is calculated using the method of Franklin and Kankanhalli [65–67] generalised to linear order. As preliminary tests of the code, we study in four dimensional phase-space the evolution of an initially small patch in a chaotic potential and the cosmological collapse of a fluctuation composed of two sinusoidal waves. We also perform a “warm” dark matter simulation in six-dimensional phase-space that we use to check the parallel scaling of the code.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Hybrid massively parallel fast sweeping method for static Hamilton–Jacobi equations

The fast sweeping method is a popular algorithm for solving a variety of static Hamilton–Jacobi equations. Fast sweeping algorithms for parallel computing have been developed, but are severely limited. In this work, we present a multilevel, hybrid parallel algorithm that combines the desirable traits of two distinct parallel methods. The fine and coarse grained components of the algorithm take advantage of heterogeneous computer architecture common in high performance computing facilities. We present the algorithm and demonstrate its effectiveness on a set of example problems including optimal control, dynamic games, and seismic wave propagation. We give results for convergence, parallel scaling, and show state-of-the-art speedup values for the fast sweeping method.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Wasatch: An architecture-proof multiphysics development environment using a Domain Specific Language and graph theory

To address the coding and software challenges of modern hybrid architectures, we propose an approach to multiphysics code development for high-performance computing. This approach is based on using a Domain Specific Language (DSL) in tandem with a directed acyclic graph (DAG) representation of the problem to be solved that allows runtime algorithm generation. When coupled with a large-scale parallel framework, the result is a portable development framework capable of executing on hybrid platforms and handling the challenges of multiphysics applications. In addition, we share our experience developing a code in such an environment – an effort that spans an interdisciplinary team of engineers and computer scientists.

97 MATHEMATICS AND COMPUTING↗

A Multiple-Set Overlapping-Domain Decomposed Monte Carlo Synthetic Acceleration Method for Linear Systems

We present a multiple-set overlapping-domain decomposed strategy for parallelizing the Monte Carlo Synthetic Acceleration method. Monte Carlo Synthetic Acceleration methods use the Neumann-Ulam class of Monte Carlo solvers for linear systems to accelerate a fixed-point iteration sequence. Effective parallel algorithms for these methods require the parallelization of the underlying Neumann-Ulam solvers. To do this in a domain decomposed environment, we borrow strategies traditionally implemented in Monte Carlo particle transport to parallelize the problem. The parallel Neumann-Ulam and multiple-set overlapping-domain decomposition algorithms are presented along with parallel scaling data for the resulting implementation using the Titan Cray XK7 machine at Oak Ridge National Laboratory.

Computer Science↗