Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “parallel, scaling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,549 records · Page 86

Collaborative Research: A Model of Partially Ionized Plasma Flows with Kinetic Treatment of Neutral Atoms and Nonthermal Ions

Interactions of flows of partially ionized, magnetized plasma are frequently accompanied by the presence of both thermal and non-thermal (pickup) ion components. Such interactions cannot be modeled using traditional MHD equations and require more advanced approaches to treat them. If a nonthermal component of ions is formed due to charge exchange and collisions between the thermal (core) ions and neutrals, it experiences the action of magnetic field, its distribution function is isotropized, and it soon acquires the velocity of the ambient plasma without being thermodynamically equilibrated. This situation, e. g., takes place in the outer heliosphere - the part of interstellar space beyond the solar system whose properties are determined by the solar wind interaction with the local interstellar medium. This is also possible in laboratory, at million degrees and above, when plasma is conducting electricity far too well, which makes Ohmic heating ineffective. To attain the target temperatures one needs additional heating eventually playing a dominant role. Among such sources is a so-called neutral particle beam heating. This is a wide-spread technique (Joint European Torus and International Thermonuclear Experimental Reactor experiments) based on the injection of powerful beams of neutral atoms into ohmically preheated plasma. In this project we have investigated the energy and density separation between the thermal and nonthermal components in the solar wind and interstellar plasmas. A new model has been developed in which we solve the ideal MHD equations for mixture of all ions and the kinetic Boltzmann equation to describe the transport of neutral atoms. As a separate capability, we can treat the flow of neutral atoms in a multi-component fashion, where neutral atoms born in each thermodynamically distinct regions are governed by the Euler gas dynamic equations. We also describe the behavior of pickup ions either kinetically, using the Fokker–Planck equation, or as a separate fluid. Our numerical simulations have demonstrated that pickup ions play a major role in the interaction of the solar wind and (partially ionized) interstellar medium plasmas. Our teams have investigated the stability of the surface (the heliopause) that separates the solar wind from the local interstellar medium, the transport of galactic cosmic rays, the properties of the heliotail flow, and modifications to the bow wave in front of the heliopause due to charge exchange between the neutral H atoms born in the solar wind and interstellar ions. Modeling results have been validated against observational data, such as obtained by the Interstellar Boundary Explorer (IBEX), and made it possible to shed light on the structure of energetic neutral atom maps created by this spacecraft.. We have also demonstrated that charge-exchange modulated heliosphere is a source of anisotropy of the multi-TeV cosmic ray flux observed in a number of Earth-bound air shower experiments. Newly developed codes are implemented within a Multi-Scale Fluid-Kinetic Simulation Suite (MS-FLUKSS), a publicly available code being developed by our team for over 12 years. MS-FLUKSS scales well up to 160,000 computing cores and has been ported on major supercomputers in the country. Efficient parallelization and data choreography in the continuum simulation modules are provided by Chombo, an adaptive mesh refinement framework managed by Phillip Colella’s team at LBNL. We have implemented in-house, hybrid (MPI+OpenMP) parallelization of the kinetic modules that solve the Boltzmann equation with a Monte Carlo method. Currently, the kinetic modules are being rewritten to take advantage of the modern CPU-GPU supercomputer architecture. The scope of the project allowed us to enhance plasma research and education in such broad, multidis- ciplinary field as physics of partially ionized plasma and its application to space physics and fusion science. Besides the impact on the modeling of complex physical systems, our approach to computational resource management for complex codes utilizing multiple algorithm technologies appears to be a major advance on current approaches. The development of sophisticated resource management will be essential for all future modeling efforts that incorporate a diversity of scales and physical processes. Our effort provided leadership in promoting computational science and plasma physics within the UAH and FIT campuses and, through the training of a broad spectrum of scientists and engineers, foster new technologies across the country.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

SPEL: Software tool for Porting E3SM Land Model with OpenACC in a Function Unit Test Framework

Most high-end computers adopt hybrid architecture, porting a large-scale scientific code onto accelerators is necessary. The paper presents a generic method for porting large-scale scientific code onto accelerators using compiler directives within a modularized function unit test platform. We have implemented the method and designed a software tool (SPEL) to port the E3SM Land Model (ELM) onto the GPUs in the Summit computer. SPEL automatically generates GPU-ready test modules for all ELM functions, such as CanopyFlux, SoilTemperature, and EcosystemDynamics. SPEL breaks the ELM into a collection of standalone unit test programs for easy code verification and further performance improvement. We further optimize several ELM test modules with advanced techniques, including memory reduction, reconstructed parallel loops, and asynchronous GPU kernel launch. We hope our study will inspire new toolkit developments that expedite large-scale scientific code porting with compiler directives.

Schwartz, Peter↗

DL_POLY Quantum 2.0: A modular general-purpose software for advanced path integral simulations

DL_POLY Quantum 2.0, a vastly expanded software based on DL_POLY Classic 1.10, is a highly parallelized computational suite written in FORTRAN77 with a modular structure for incorporating nuclear quantum effects into large-scale/long-time molecular dynamics simulations. This is achieved by presenting users with a wide selection of state-of-the-art dynamics methods that utilize the isomorphism between a classical ring polymer and Feynman’s path integral formalism of quantum mechanics. Here, the flexible and user-friendly input/output handling system allows the control of methodology, integration schemes, and thermostatting. DL_POLY Quantum is equipped with a module specifically assigned for calculating correlation functions and printing out the values for sought-after quantities, such as dipole moments and center-of-mass velocities, with packaged tools for calculating infrared absorption spectra and diffusion coefficients.

36 MATERIALS SCIENCE↗

EXAMINING SUBGRID MODELS OF SUPERMASSIVE BLACK HOLES IN COSMOLOGICAL SIMULATION

Although supermassive black holes (SMBHs) play an important role in galaxy and cluster evolution, at present they can only be included in large-scale cosmological simulation via subgrid techniques. However, these subgrid models have not been studied in a systematic fashion. Using a newly developed fast, parallel spherical overdensity halo finder built into the simulation code FLASH, we perform a suite of dark matter-only cosmological simulations to study the effects of subgrid model choice on relations between SMBH mass and dark matter halo mass and velocity dispersion. We examine three aspects of SMBH subgrid models: the choice of initial black hole seed mass, the test for merging two black holes, and the frequency of applying the subgrid model. We also examine the role that merging can play in determining the relations, ignoring the complicating effects of SMBH-driven accretion and feedback. We find that the choice of subgrid model can dramatically affect the black hole merger rate, the cosmic SMBH mass density, and the low-redshift relations to halo properties. We also find that it is possible to reproduce observations of the low-redshift relations without accretion and feedback, depending on the choice of subgrid model.

79 ASTRONOMY AND ASTROPHYSICS↗

Lazy Shadowing: An Adaptive, Power-Aware, Resiliency Framework for Extreme Scale Computing

The inherent instability of extreme-scale computing systems puts into question the viability of traditional fault-tolerance methods and calls for a reconsideration of the fault-tolerance and power-awareness problem, at scale. This project explores a novel paradigm, Lazy Shadowing, to achieve high resiliency in future extreme-scale systems. The basic tenet of Lazy Shadowing is the association of a shadow with each process which runs in parallel, but on a different node and at a reduced execution rate, than its associated process. The successful completion of the main process causes the immediate termination of the shadow, resulting in significant energy savings. Upon failure of the main process, however, the shadow’s execution rate is increased so that it completes by the expected completion time without significantly impacting the progress of the remaining tasks. Different approaches can be used to control execution rate, including co-locating multiple shadows on a single computing node and/or using Dynamic Voltage and Frequency Scaling.

97 MATHEMATICS AND COMPUTING↗

MASSIVELY PARALLEL LATENT SEMANTIC ANALYSES USING A GRAPHICS PROCESSING UNIT

Latent Semantic Analysis (LSA) aims to reduce the dimensions of large term-document datasets using Singular Value Decomposition. However, with the ever-expanding size of datasets, current implementations are not fast enough to quickly and easily compute the results on a standard PC. A graphics processing unit (GPU) can solve some highly parallel problems much faster than a traditional sequential processor or central processing unit (CPU). Thus, a deployable system using a GPU to speed up large-scale LSA processes would be a much more effective choice (in terms of cost/performance ratio) than using a PC cluster. Due to the GPU’s application-specifi c architecture, harnessing the GPU’s computational prowess for LSA is a great challenge. We presented a parallel LSA implementation on the GPU, using NVIDIA® Compute Unifi ed Device Architecture and Compute Unifi ed Basic Linear Algebra Subprograms software. The performance of this implementation is compared to traditional LSA implementation on a CPU using an optimized Basic Linear Algebra Subprograms library. After implementation, we discovered that the GPU version of the algorithm was twice as fast for large matrices (1 000x1 000 and above) that had dimensions not divisible by 16. For large matrices that did have dimensions divisible by 16, the GPU algorithm ran fi ve to six times faster than the CPU version. The large variation is due to architectural benefi ts of the GPU for matrices divisible by 16. It should be noted that the overall speeds for the CPU version did not vary from relative normal when the matrix dimensions were divisible by 16. Further research is needed in order to produce a fully implementable version of LSA. With that in mind, the research we presented shows that the GPU is a viable option for increasing the speed of LSA, in terms of cost/performance ratio.

Cavanagh, J↗

High density Langmuir probe array for NSTX scrape-off layer measurements under lithiated divertor conditions

A high density Langmuir probe array has been developed for measurements of scrape-off layer parameters in NSTX. Relevant scale lengths for heat and particle fluxes are 1-5 cm. Transient edge plasma events can occur on a time scale of several milliseconds, and the duration of a typical plasma discharge is {approx}1 s. The array consists of 99 individual electrodes arranged in three parallel radial rows to allow both swept and triple-probe operation and is mounted in a carbon tile located in the lower outer divertor of NSTX between two segments of the newly installed liquid lithium divertor. Initial swept probe results tracking the outer strike point through probe flux measurements are presented.

36 MATERIALS SCIENCE↗

Fast electrostatic force calculation on parallel computer clusters

The fast multipole method (FMM) and smooth particle mesh Ewald (SPME) are well known fast algorithms to evaluate long range electrostatic interactions in molecular dynamics and other fields. FMM is a multi-scale method which reduces the computation cost by approximating the potential due to a group of particles at a large distance using few multipole functions. This algorithm scales like O(N) for N particles. SPME algorithm is an O(NlnN) method which is based on an interpolation of the Fourier space part of the Ewald sum and evaluating the resulting convolutions using fast Fourier transform (FFT). Those algorithms suffer from relatively poor efficiency on large parallel machines especially for mid-size problems around hundreds of thousands of atoms. A variation of the FMM, called PWA, based on plane wave expansions is presented in this paper. A new parallelization strategy for PWA, which takes advantage of the specific form of this expansion, is described. Its parallel efficiency is compared with SPME through detail time measurements on two different computer clusters.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Hierarchical fractional-step approximations and parallel kinetic Monte Carlo algorithms

We present a mathematical framework for constructing and analyzing parallel algorithms for lattice kinetic Monte Carlo (KMC) simulations. The resulting algorithms have the capacity to simulate a wide range of spatio-temporal scales in spatially distributed, non-equilibrium physiochemical processes with complex chemistry and transport micro-mechanisms. Rather than focusing on constructing exactly the stochastic trajectories, our approach relies on approximating the evolution of observables, such as density, coverage, correlations and so on. More specifically, we develop a spatial domain decomposition of the Markov operator (generator) that describes the evolution of all observables according to the kinetic Monte Carlo algorithm. This domain decomposition corresponds to a decomposition of the Markov generator into a hierarchy of operators and can be tailored to specific hierarchical parallel architectures such as multi-core processors or clusters of Graphical Processing Units (GPUs). Based on this operator decomposition, we formulate parallel Fractional step kinetic Monte Carlo algorithms by employing the Trotter Theorem and its randomized variants; these schemes, (a) are partially asynchronous on each fractional step time-window, and (b) are characterized by their communication schedule between processors. The proposed mathematical framework allows us to rigorously justify the numerical and statistical consistency of the proposed algorithms, showing the convergence of our approximating schemes to the original serial KMC. The approach also provides a systematic evaluation of different processor communicating schedules. We carry out a detailed benchmarking of the parallel KMC schemes using available exact solutions, for example, in Ising-type systems and we demonstrate the capabilities of the method to simulate complex spatially distributed reactions at very large scales on GPUs. Finally, we discuss work load balancing between processors and propose a re-balancing scheme based on probabilistic mass transport methods.

97 MATHEMATICS AND COMPUTING↗

The Effective Fragment Molecular Orbital Method: Achieving High Scalability and Accuracy for Large Systems

The effective fragment molecular orbital (EFMO) method has been developed to predict the total energy of a very large molecular system accurately (with respect to the underlying quantum mechanical method) and efficiently by taking advantage of the locality of strong chemical interactions and employing a two-level hierarchical parallelism. The accuracy of the EFMO method is partly attributed to the accurate and robust intermolecular interaction prediction between distant fragments, in particular, the many-body polarization and dispersion effects, which require the generation of static and dynamic polarizability tensors by solving the coupled perturbed Hartree–Fock (CPHF) and time-dependent HF (TDHF) equations, respectively. Solving the CPHF and TDHF equations is the main EFMO computational bottleneck due to the inefficient (serial) and I/O-intensive implementation of the CPHF and TDHF solvers. In this work, the efficiency and scalability of the EFMO method are significantly improved with a new CPU memory-based implementation for solving the CPHF and TDHF equations that are parallelized by either message passing interface (MPI) or hybrid MPI/OpenMP. Here, the accuracy of the EFMO method is demonstrated for both covalently bonded systems and noncovalently bound molecular clusters by systematically examining the effects of basis sets and a key distance-related cutoff parameter, R cut . R cut determines whether a fragment pair (dimer) is treated by the chosen ab initio method or calculated using the effective fragment potential (EFP) method (separated dimers). Decreasing the value of Rcut increases the number of separated (EFP) dimers, thereby decreasing the computational effort. It is demonstrated that excellent accuracy (<1 kcal/mol error per fragment) can be achieved when using a sufficiently large basis set with diffuse functions coupled with a small R cut value. With the new parallel implementation, the total EFMO wall time is substantially reduced, especially with a high number of MPI ranks. Given a sufficient workload, nearly ideal strong scaling is achieved for the CPHF and TDHF parts of the calculation. For the first time, EFMO calculations with the inclusion of long-range polarization and dispersion interactions on a hydrated mesoporous silica nanoparticle with explicit water solvent molecules (more than 15k atoms) are achieved on a massively parallel supercomputer using nearly 1000 physical nodes. In addition, EFMO calculations on the carbinolamine formation step of an amine-catalyzed aldol reaction at the nanoscale with explicit solvent effects are presented.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Ion and Electron Acoustic Bursts during Anti-Parallel Reconnection Driven by Lasers

Magnetic reconnection converts magnetic energy into thermal and kinetic energy in plasma. Among the numerous candidate mechanisms, ion acoustic instabilities driven by the relative drift between ions and electrons (or equivalently, electric current) have been suggested to play a critical role in dissipating magnetic energy in collisionless plasmas. However, their existence and effectiveness during reconnection have not been well understood due to ion Landau damping and difficulties in resolving the Debye length scale in the laboratory. We report a sudden onset of ion acoustic bursts measured by collective Thomson scattering in the exhaust of anti-parallel magnetically driven reconnection using high-power lasers. The ion acoustic bursts are followed by electron acoustic bursts with electron heating and bulk acceleration. We reproduce these observations with one- and two-dimensional particle-in-cell simulations in which an electron outflow jet drives ion acoustic instabilities, forming double layers. These layers induce electron two-stream instabilities that generate electron acoustic bursts and energize electrons. Our results demonstrate the importance of ion and electron acoustic dynamics during reconnection when ion Landau damping is ineffective, a condition applicable to a range of astrophysical plasmas including near-Earth space, stellar flares and black hole accretion engines.

70 PLASMA PHYSICS AND FUSION TECHNOLOGY↗

Parallel Distributed Hydrology Soil Vegetation Model (DHSVM) using global arrays

The Distributed Hydrology Soil Vegetation Model (DHSVM) code was parallelized for distributed memory computers using Global Arrays (GA). To analyze parallel performance, DHSVM was used to simulate two significantly sized river basins located in northwest continental United States and southwest Canada at 90~m resolution: (1) Clearwater (25,000~km2) and (2) Columbia (668,000~km2). Meteorological forcing applied to both basins was dynamically down-scaled from a global circulation model using the Weather Research and Forecasting (WRF) model and read into DHSVM as 2D maps each time step. We report parallel code speed up was significant. Run times for 1-year simulations were reduced by an order of magnitude for both test basins. A maximum speed up of 105 was attained with 480 processors while simulating the Columbia basin. Speed up was limited by input dominated tasks, particularly the input of meteorological forcing data.

54 ENVIRONMENTAL SCIENCES↗

Fabrication of First 4-m Coils for the LARP MQXFA Quadrupole and Assembly in Mirror Structure

The US LHC Accelerator Research Program is constructing prototype interaction region quadrupoles as part of the US in-kind contribution to the Hi-Lumi LHC project. The low-beta MQXFA Q1/Q3 coils have a 4-m length and a 150 mm bore. The design is first validated on short, one meter models (MQXFS) developed as part of the longstanding Nb3Sn quadrupole R&D by LARP in collaboration with CERN. In parallel, facilities and tooling are being developed and refined at BNL, LBNL, and FNAL to enable long coil production, assembly, and cold testing. Long length scale-up is based on the experience from the LARP 90 mm aperture (TQ-LQ) and 120 mm aperture (HQ and Long HQ) programs. A 4-m long MQXF practice coil was fabricated, water jet cut and analyzed to verify procedures, parts, and tooling. In parallel, the first complete prototype coil (QXFP01a) was fabricated and assembled in a long magnetic mirror, MQXFPM1, to provide early feedback on coil design and fabrication following the successful experience of previous LARP mirror tests.

43 PARTICLE ACCELERATORS↗

Drift Kinetic Theory and Cosmic Rays

Starting from the Vlasov or Boltzmann equation for cosmic rays in a random and regular magnetic field, we introduce guiding center coordinates and transform the velocity to a frame moving at the electric field drift velocity. The resultant equation is written in terms of the parallel and perpendicular momentum and gyro-phase of the particle, and describes spatial particle transport in guiding center coordinates. Using the drift ordering in which the gyro-scale and gyro-period are assumed short compared to the background flow length and time scales, and averaging over the gyro-phase gives the drift kinetic equation in which the adiabatic moment and total particle energy in the inertial frame are used to describe the momentum and energy of the particle. If the parallel electric field is small, the adiabatic moment of the particles is conserved to lowest order in the drift ordering. The resultant drift kinetic equation properly takes into account the energy changes of the particles due to drifts along the electric field, and betatron acceleration, but contains only the lowest order approximation for the guiding center drift velocity to describe the spatial advection of the particles. A further transformation of variables, in which the particle momentum and pitch angle are specified in the local fluid frame, gives the focussed transport equation derived by Skilling to describe particle transport in a moving plasma medium, such as the solar wind. The connections to previous derivations of the Skilling's pitch angle focussed transport equation are discussed.

79 ASTRONOMY AND ASTROPHYSICS↗

High-performance epistasis detection in quantitative trait GWAS

epiSNP is a program for identifying pairwise single nucleotide polymorphism (SNP) interactions (epistasis) in quantitative-trait genome-wide association studies (GWAS). A parallel MPI version (EPISNPmpi) was created in 2008 to address this computationally expensive analysis on large data sets with many quantitative traits and SNP markers. However, the falling cost of genotyping has led to an explosion of large-scale GWAS data sets that challenge EPISNPmpi’s ability to compute results in a reasonable amount of time. Therefore, we optimized epiSNP for modern multi-core and highly parallel many-core processors to efficiently handle these large data sets. This paper describes the serial optimizations, dynamic load balancing using MPI-3 RMA operations, and shared-memory parallelization with OpenMP to further enhance load balancing and allow execution on the Intel Xeon Phi coprocessor (MIC). For a large GWAS data set, our optimizations provided a 38.43× speedup over EPISNPmpi on 126 nodes using 2 MICs on TACC’s Stampede Supercomputer. We also describe a Coarray Fortran (CAF) version that demonstrates the suitability of PGAS languages for problems with this computational pattern. We show that the Coarray version performs competitively with the MPI version on the NERSC Edison Cray XC30 supercomputer. Finally, the performance benefits of hyper-threading for this application on Edison (average 1.35× speedup) are demonstrated.

97 MATHEMATICS AND COMPUTING↗

A Parallel Implementation of a Smoothed Particle Hydrodynamics Method on Graphics Hardware Using the Compute Unified Device Architecture

The smoothed particle hydrodynamics (SPH), which is a class of meshfree particle methods (MPMs), has a wide range of applications from micro-scale to macro-scale as well as from discrete systems to continuum systems. Graphics hardware, originally designed for computer graphics, now provide unprecedented computational power for scientific computation. Particle system needs a huge amount of computations in physical simulation. In this paper, an efficient parallel implementation of a SPH method on graphics hardware using the Compute Unified Device Architecture is developed for fluid simulation. Comparing to the corresponding CPU implementation, our experimental results show that the new approach allows significant speedups of fluid simulation through handling huge amount of computations in parallel on graphics hardware.

97 MATHEMATICS AND COMPUTING↗

Machine Learning Based Parallel I/O Predictive Modeling: A Case Study on Lustre File Systems

Parallel I/O hardware and software infrastructure is a key contributor to performance variability for applications running on large-scale HPC systems. This variability confounds efforts to predict application performance for characterization, modeling, optimization, and job scheduling. We propose a modeling approach that improves predictive ability by explicitly treating the variability and by leveraging the sensitivity of application parameters on performance to group applications with similar characteristics. We develop a Gaussian process-based machine learning algorithm to model I/O performance and its variability as a function of application and file system characteristics. We demonstrate the effectiveness of the proposed approach using data collected from the Edison system at the National Energy Research Scientific Computing Center. The results show that the proposed sensitivity-based models are better at prediction when compared with application-partitioned or unpartitioned models. We highlight modeling techniques that are robust to the outliers that can occur in production parallel file systems. Using the developed metrics and modeling approach, we provide insights into the file system metrics that have a significant impact on I/O performance.

I/O performance variability↗

Scalable training of trustworthy and energy-efficient predictive graph foundation models for atomistic materials modeling: a case study with HydraGNN

We present our work on developing and training scalable, trustworthy, and energy-efficient predictive graph foundation models (GFMs) using HydraGNN, a multi-headed graph convolutional neural network architecture. HydraGNN expands the boundaries of graph neural network (GNN) computations in both training scale and data diversity. It abstracts over message passing algorithms, allowing both reproduction of and comparison across algorithmic innovations that define nearest-neighbor convolution in GNNs. This work discusses a series of optimizations that have allowed scaling up the GFMs training to tens of thousands of GPUs on datasets consisting of hundreds of millions of graphs. Our GFMs use multitask learning (MTL) to simultaneously learn graph-level and node-level properties of atomistic structures, such as energy and atomic forces. Using over 154 million atomistic structures for training, we illustrate the performance of our approach along with the lessons learned on two state-of-the-art US Department of Energy (US-DOE) supercomputers, namely the Perlmutter petascale system at the National Energy Research Scientific Computing Center and the Frontier exascale system at Oak Ridge Leadership Computing Facility. The HydraGNN architecture enables the GFM to achieve near-linear strong scaling performance using more than 2000 GPUs on Perlmutter and 16,000 GPUs on Frontier.

97 MATHEMATICS AND COMPUTING↗