Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “parallel, scaling”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

Sweepless time-dependent transport calculations using the staggered block Jacobi method

The Staggered-Block Jacobi (SBJ) method is a new numerical SN transport method for solving time-dependent problems without sweeps or low-order acceleration. Because it is a Jacobian method, it is trivial to parallelize and will scale linearly with the number of processors, It is highly accurate in thick-diffusive problems and unconditionally stable when combined with the lumped linear discontinuous finite element spatial discretization. In this way, the SBJ method is complementary to sweep-based methods, which are accurate and efficient in thin, streaming regions but inefficient in thick, diffusive problems without acceleration. We have extended previous work by demonstrating how sweep-based methods and the SBJ method may be combined to produce a method which is accurate and efficient without acceleration in all optical thicknesses while still retaining good parallel efficiency. Furthermore, iterations may also be added to the SBJ method. This is particularly useful for improving the accuracy of the SBJ method in intermediate-thickness problems.

97 MATHEMATICS AND COMPUTING↗

TuckerMPI: A Parallel C++/MPI Software Package for Large-scale Data Compression via the Tucker Tensor Decomposition

With this study, our goal is compression of massive-scale grid-structured data, such as the multi-terabyte output of a high-fidelity computational simulation. For such data sets, we have developed a new software package called TuckerMPI, a parallel C++/MPI software package for compressing distributed data. The approach is based on treating the data as a tensor, i.e., a multidimensional array, and computing its truncated Tucker decomposition, a higher-order analogue to the truncated singular value decomposition of a matrix. The result is a low-rank approximation of the original tensor-structured data. Compression efficiency is achieved by detecting latent global structure within the data, which we contrast to most compression methods that are focused on local structure. In this work, we describe TuckerMPI, our implementation of the truncated Tucker decomposition, including details of the data distribution and in-memory layouts, the parallel and serial implementations of the key kernels, and analysis of the storage, communication, and computational costs. We test the software on 4.5 and 6.7 terabyte data sets distributed across 100 s of nodes (1,000 s of MPI processes), achieving compression ratios between 100 and 200,000×, which equates to 99--99.999% compression (depending on the desired accuracy) in substantially less time than it would take to even read the same dataset from a parallel file system. Moreover, we show that our method also allows for reconstruction of partial or down-sampled data on a single node, without a parallel computer so long as the reconstructed portion is small enough to fit on a single machine, e.g., in the instance of reconstructing/visualizing a single down-sampled time step or computing summary statistics. The code is available at https://gitlab.com/tensors/TuckerMPI.

97 MATHEMATICS AND COMPUTING↗

Graph-based quantum response theory and shadow Born–Oppenheimer molecular dynamics

Graph-based linear scaling electronic structure theory for quantum-mechanical molecular dynamics simulations [A. M. N. Niklasson et al., J. Chem. Phys. 144, 234101 (2016)] is adapted to the most recent shadow potential formulations of extended Lagrangian Born–Oppenheimer molecular dynamics, including fractional molecular-orbital occupation numbers [A. M. N. Niklasson, J. Chem. Phys. 152, 104103 (2020) and A. M. N. Niklasson, Eur. Phys. J. B 94, 164 (2021)], which enables stable simulations of sensitive complex chemical systems with unsteady charge solutions. The proposed formulation includes a preconditioned Krylov subspace approximation for the integration of the extended electronic degrees of freedom, which requires quantum response calculations for electronic states with fractional occupation numbers. For the response calculations, we introduce a graph-based canonical quantum perturbation theory that can be performed with the same natural parallelism and linear scaling complexity as the graph-based electronic structure calculations for the unperturbed ground state. Further, the proposed techniques are particularly well-suited for semi-empirical electronic structure theory, and the methods are demonstrated using self-consistent charge density-functional tight-binding theory both for the acceleration of self-consistent field calculations and for quantum-mechanical molecular dynamics simulations. Graph-based techniques combined with the semi-empirical theory enable stable simulations of large, complex chemical systems, including tens-of-thousands of atoms.

74 ATOMIC AND MOLECULAR PHYSICS↗

Optimization of Sparse Matrix-Vector Multiplication on Emerging Multicore Platforms

We are witnessing a dramatic change in computer architecture due to the multicore paradigm shift, as every electronic device from cell phones to supercomputers confronts parallelism of unprecedented scale. To fully unleash the potential of these systems, the HPC community must develop multicore specific-optimization methodologies for important scientific computations. In this work, we examine sparse matrix-vector multiply (SpMV) - one of the most heavily used kernels in scientific computing - across a broad spectrum of multicore designs. Our experimental platform includes the homogeneous AMD quad-core, AMD dual-core, and Intel quad-core designs, the heterogeneous STI Cell, as well as one of the first scientific studies of the highly multithreaded Sun Victoria Falls (a Niagara2 SMP). We present several optimization strategies especially effective for the multicore environment, and demonstrate significant performance improvements compared to existing state-of-the-art serial and parallel SpMV implementations. Additionally, we present key insights into the architectural trade-offs of leading multicore design strategies, in the context of demanding memory-bound numerical algorithms.

97 MATHEMATICS AND COMPUTING↗

Goethite Bench-scale and Large-scale Preparation Tests

The Hanford Waste Treatment and Immobilization Plant (WTP) is the keystone for cleanup of high-level radioactive waste from our nation's nuclear defense program. The WTP will process high-level waste from the Hanford tanks and produce immobilized high-level waste glass for disposal at a national repository, low activity waste (LAW) glass, and liquid effluent from the vitrification off-gas scrubbers. The liquid effluent will be stabilized into a secondary waste form (e.g. grout-like material) and disposed on the Hanford site in the Integrated Disposal Facility (IDF) along with the low-activity waste glass. The major long-term environmental impact at Hanford results from technetium that volatilizes from the WTP melters and finally resides in the secondary waste. Laboratory studies have indicated that pertechnetate ({sup 99}TcO{sub 4}{sup -}) can be reduced and captured into a solid solution of {alpha}-FeOOH, goethite (Um 2010). Goethite is a stable mineral and can significantly retard the release of technetium to the environment from the IDF. The laboratory studies were conducted using reaction times of many days, which is typical of environmental subsurface reactions that were the genesis of this new process. This study was the first step in considering adaptation of the slow laboratory steps to a larger-scale and faster process that could be conducted either within the WTP or within the effluent treatment facility (ETF). Two levels of scale-up tests were conducted (25x and 400x). The largest scale-up produced slurries of Fe-rich precipitates that contained rhenium as a nonradioactive surrogate for {sup 99}Tc. The slurries were used in melter tests at Vitreous State Laboratory (VSL) to determine whether captured rhenium was less volatile in the vitrification process than rhenium in an unmodified feed. A critical step in the technetium immobilization process is to chemically reduce Tc(VII) in the pertechnetate (TcO{sub 4}{sup -}) to Tc(Iv)by reaction with the ferrous ion, Fe{sup 2+}-Fe{sup 2+} is oxidized to Fe{sup 3+} - in the presence of goethite seed particles. Rhenium does not mimic that process; it is not a strong enough reducing agent to duplicate the TcO{sub 4}{sup -}/Fe{sup 2+} redox reactions. Laboratory tests conducted in parallel with these scaled tests identified modifications to the liquid chemistry necessary to reduce ReO{sub 4}{sup -} and capture rhenium in the solids at levels similar to those achieved by Um (2010) for inclusion of Tc into goethite. By implementing these changes, Re was incorporated into Fe-rich solids for testing at VSL. The changes also changed the phase of iron that was in the slurry product: rather than forming goethite ({alpha}-FeOOH), the process produced magnetite (Fe{sub 3}O{sub 4}). Magnetite was considered by Pacific Northwest National Laboratory (PNNL) and VSL to probably be a better product to improve Re retention in the melter because it decomposes at a higher temperature than goethite (1538 C vs. 136 C). The feasibility tests at VSL were conducted using Re-rich magnetite. The tests did not indicate an improved retention of Re in the glass during vitrification, but they did indicate an improved melting rate (+60%), which could have significant impact on HLW processing. It is still to be shown whether the Re is a solid solution in the magnetite as {sup 99}Tc was determined to be in goethite.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Accelerated Electrons in the LLBL as Observed by Interball on February 15, 1996

We analyze electron observations in the low latitude boundary layer (LLBL) with the Interball Tail satellite under a southward-duskward magnetosheath magnetic field. It is well known that the LLBL is characterized by bi-directional electron velocity distributions, which leads to an enhancement of the electron temperature. It is also well known that the electron temperature within the LLBL increases with a decrease of the LLBL number density. We found that the parallel electron velocity distribution changes self-consistently with varying number density. Namely, the shape of the parallel velocity distribution does not change in a log-log scale indicating that electron parallel velocities increase proportionately as the number density decreases. This suggests a possible role for the Fermi acceleration mechanism. We also discuss the LLBL topology that is provided by electron velocity distributions.

Vaisberg, O. L.↗

GPU acceleration of all-electron electronic structure theory using localized numeric atom-centered basis functions

We present an implementation of all-electron density-functional theory for massively parallel GPU-based platforms, using localized atom-centered basis functions and real-space integration grids. Special attention is paid to domain decomposition of the problem on non-uniform grids, which enables compute- and memory-parallel execution across thousands of nodes for real-space operations, e.g. the update of the electron density, the integration of the real-space Hamiltonian matrix, and calculation of Pulay forces. To assess the performance of our GPU implementation, we performed benchmarks on three different architectures using a 103-material test set. We find that operations which rely on dense serial linear algebra show dramatic speedups from GPU acceleration: in particular, SCF iterations including force and stress calculations exhibit speedups ranging from 4.5 to 6.6. For the architectures and problem types investigated here, this translates to an expected overall speedup between 3–4 for the entire calculation (including non-GPU accelerated parts), for problems featuring several tens to hundreds of atoms. Additional calculations for a 375-atom Bi2Se3 bilayer show that the present GPU strategy scales for large-scale distributed-parallel simulations.

42 ENGINEERING↗

Dynamic Load Balancing for Finite Element Calculations on Parallel Computers

Computational requirements of full scale computational fluid dynamics change as computation progresses on a parallel machine. The change in computational intensity causes workload imbalance of processors, which in turn requires a large amount of data movement at runtime. If parallel CFD is to be successful on a parallel or massively parallel machine, balancing of the runtime load is indispensable. Here a frame work is presented for dynamic load balancing for CFD applications, called Jove. One processor is designated as a decision maker Jove while others are assigned to computational fluid dynamics. Processors running CFD send flags to Jove in a predetermined number of iterations to initiate load balancing. Jove starts working on load balancing while other processors continue working with the current data and load distribution. Jove goes through several steps to decide if the new data should be taken, including preliminary evaluate, partition, processor reassignment, cost evaluation, and decision. Jove running on a single SP2 node has been completely implemented. Preliminary experimental results show that the Jove approach to dynamic load balancing can be effective for full scale grid partitioning on the target machine SP2.

Pramono, Eddy↗

Transferring predictions of formation energy across lattices of increasing size*

In this study, we show the transferability of graph convolutional neural network (GCNN) predictions of the formation energy of the nickel-platinum solid solution alloy across atomic structures of increasing sizes. The original dataset was generated with the large-scale atomic/molecular massively parallel simulator using the second nearest-neighbor modified embedded-atom method empirical interatomic potential. Geometry optimization was performed on the initially randomly generated face centered cubic crystal structures and the formation energy has been calculated at each step of the geometry optimization, with configurations spanning the whole compositional range. Using data from various steps of the geometry optimization, we first trained our open-source, scalable implementation of GCNN called HydraGNN on a lattice of 256 atoms, which accounts well for the short-range interactions. Using this data, we predicted the formation energy for lattices of 864 atoms and 2048 atoms, which resulted in lower-than-expected accuracy due to the long-range interactions present in these larger lattices. We accounted for the long-range interactions by including a small amount of training data representative for those two larger sizes, whereupon the predictions of HydraGNN scaled linearly with the size of the lattice. Therefore, our strategy ensured scalability while reducing significantly the computational cost of training on larger lattice sizes.

36 MATERIALS SCIENCE↗

Non-equilibrium reaction and relaxation dynamics in a strongly interacting explicit solvent: F + CD{sub 3}CN treated with a parallel multi-state EVB model

We describe a parallelized linear-scaling computational framework developed to implement arbitrarily large multi-state empirical valence bond (MS-EVB) calculations within CHARMM and TINKER. Forces are obtained using the Hellmann-Feynman relationship, giving continuous gradients, and good energy conservation. Utilizing multi-dimensional Gaussian coupling elements fit to explicitly correlated coupled cluster theory, we built a 64-state MS-EVB model designed to study the F + CD{sub 3}CN → DF + CD{sub 2}CN reaction in CD{sub 3}CN solvent (recently reported in Dunning et al. [Science 347(6221), 530 (2015)]). This approach allows us to build a reactive potential energy surface whose balanced accuracy and efficiency considerably surpass what we could achieve otherwise. We ran molecular dynamics simulations to examine a range of observables which follow in the wake of the reactive event: energy deposition in the nascent reaction products, vibrational relaxation rates of excited DF in CD{sub 3}CN solvent, equilibrium power spectra of DF in CD{sub 3}CN, and time dependent spectral shifts associated with relaxation of the nascent DF. Many of our results are in good agreement with time-resolved experimental observations, providing evidence for the accuracy of our MS-EVB framework in treating both the solute and solute/solvent interactions. The simulations provide additional insight into the dynamics at sub-picosecond time scales that are difficult to resolve experimentally. In particular, the simulations show that (immediately following deuterium abstraction) the nascent DF finds itself in a non-equilibrium regime in two different respects: (1) it is highly vibrationally excited, with ∼23 kcal mol{sup −1} localized in the stretch and (2) its post-reaction solvation environment, in which it is not yet hydrogen-bonded to CD{sub 3}CN solvent molecules, is intermediate between the non-interacting gas-phase limit and the solution-phase equilibrium limit. Vibrational relaxation of the nascent DF results in a spectral blue shift, while relaxation of the post-reaction solvation environment results in a red shift. These two competing effects mean that the post-reaction relaxation profile is distinct from what is observed when Franck-Condon vibrational excitation of DF occurs within a microsolvation environment initially at equilibrium. Our conclusions, along with the theoretical and parallel software framework presented in this paper, should be more broadly applicable to a range of complex reactive systems.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Physical test of a particle simulation model in a sheared granular system

We report a detailed comparison of a slow gravity driven sheared granular flow with a computational model performed with the Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS). To our knowledge, this is the first thorough test of the LAMMPS model with a laboratory granular flow. In the experiments, grains flow inside a silo with a rectangular cross-section, and are sheared by a rough boundary on one side and smooth boundaries on the other sides. Individual grain position and motion are measured using a particle index matching imaging technique where a fluorescent dye is added to the interstitial liquid which has the same refractive index as the glass beads. The boundary imposes a packing order, and the grains are observed to flow in layers which get progressively more disordered with distance from the walls. The computations use a Cundall--Strack contact model between the grains, using contact parameters that have been used in many other previous studies, and ignore the hydrodynamic effects of the interstitial liquid. Computations are performed to understand the effect of particle coefficient of friction, elasticity, contact model, and polydispersity on mean flow properties. After appropriate scaling, we find that the mean velocity of the grains and the number density as a function of flow cross-section observed in the experiments and the simulations are in excellent agreement. The mean flow profile is observed to be unchanged over a broad range of coefficient of friction, except near the smooth wall. We show that the flow profile is not sensitive to atleast 10\percent polydispersity in particle size. Because the grain elasticity used is smaller in the computations as compared with glass grains, wave-like features can be noted over short time scales in the mean velocity and the velocity auto-correlations measured in the simulations. These wave features occur over an intermediate timescale larger than the particle interaction but smaller than the timescale of the macroscopic flow features. The wave features become more prominent as grain elasticity is further reduced. We then perform a detailed comparison of the particle fluctuation properties as measured by the displacement probability distribution function and the mean square displacement. Excellent agreement is observed over a time interval over which particles can be tracked effectively in the experiments. Using the longer tracking intervals possible in the simulations, we find that the diffusion inthe layers is greater in the flow direction, than in the perpendicular direction. Further signatures of confinement and hopping between layers is observed. All in all, our study provides strong support for the LAMMPS model of granular flow, and further supports the hypothesis that the interstitial liquid has negligible effects on granular fluctuations provided the flow is slow.

36 MATERIALS SCIENCE↗

Using Parameter Sweep in WaterTAP to Analyze New Water Treatment Technologies

We describe a powerful and generalized parameter sweep tool in this report that was originally developed to analyze the performance of existing and novel water treatment models being developed in WaterTAP. Since WaterTAP is built upon IDAES and Pyomo, the parameter sweep tool can be used to systematically explore and debug the behavior of most Pyomo and IDAES numerical models. In order to enable meaningful analyses, the parameter sweep tool has been designed with the following features: 1) Model flexibility: The parameter sweep tool does not enforce any restrictions on the types of models that can be used with it. As long as a Pyomo model can be solved and the parameter is active and mutable, the tool only needs functions that describe how to run the model, the sweep parameters, and the output quantities of interest. 2) Flexible sampling: The parameter sweep tool has inbuilt functions to generate samples from a random distribution or a multidimensional Euclidean space. Furthermore, the users have to ability to supply samples generated from a tool of their choice. 3) Multiple sweep types: A user can choose from one of 3 types of parameter sweeps depending on their needs. 4) Detailed outputs: Outputs generated by the parameter sweep tool can be stored in detailed H5 file or user-friendly CSV files for post processing. 5) Parallel computing: The parameter sweep supports shared and distributed memory parallel computing to enable the use of high performance computers (HPC) for large-scale analyses. 6) Modular: The parameter sweep tool is self-contained and can easily be integrated within an outer-loop analysis or as desired by the user. 7) Ease of use: The tool is well documented and a simple sweep can be easily executed by following the online documentation in a few lines of code. We demonstrate the use of the parameter sweep tool on a simple water treatment system from the WaterTAP repository and show its parallel scaling performance on an Apple laptop and NREL's Eagle HPC. The parameter sweep tool is actively being used with models currently being developed within WaterTAP and we expect its use to grow beyond it to other IDAES and Pyomo models.

97 MATHEMATICS AND COMPUTING↗

UPC++ Programmer’s Guide (v1.0-2019.3.0)

UPC++ is a C++11 library that provides Partitioned Global Address Space (PGAS) programming. It is designed for writing parallel programs that run efficiently and scale well on distributed-memory parallel computers. The PGAS model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. However, PGAS also provides access to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ provides numerous methods for accessing and using global memory. In UPC++, all operations that access remote memory are explicit, which encourages programmers to be aware of the cost of communication and data movement. Moreover, all remote-memory access operations are by default asynchronous, to enable programmers to write code that scales well even on hundreds of thousands of cores.

97 MATHEMATICS AND COMPUTING↗

UPC++ v1.0 Programmer’s Guide (Revision 2019.9.0)

UPC++ is a C++11 library that provides Partitioned Global Address Space (PGAS) programming. It is designed for writing parallel programs that run efficiently and scale well on distributed-memory parallel computers. The PGAS model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. However, PGAS also provides access to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ provides numerous methods for accessing and using global memory. In UPC++, all operations that access remote memory are explicit, which encourages programmers to be aware of the cost of communication and data movement. Moreover, all remote-memory access operations are by default asynchronous, to enable programmers to write code that scales well even on hundreds of thousands of cores.

97 MATHEMATICS AND COMPUTING↗

UPC++ v1.0 Programmer’s Guide, Revision 2020.10.0

UPC++ is a C++11 library that provides Partitioned Global Address Space (PGAS) programming. It is designed for writing parallel programs that run efficiently and scale well on distributed-memory parallel computers. The PGAS model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. However, PGAS also provides access to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ provides numerous methods for accessing and using global memory. In UPC++, all operations that access remote memory are explicit, which encourages programmers to be aware of the cost of communication and data movement. Moreover, all remote-memory access operations are by default asynchronous, to enable programmers to write code that scales well even on hundreds of thousands of cores.

97 MATHEMATICS AND COMPUTING↗

UPC++ Programmer’s Guide, v1.0-2018.9.0

UPC++ is a C++11 library that provides Partitioned Global Address Space (PGAS) programming. It is designed for writing parallel programs that run efficiently and scale well on distributed-memory parallel computers. The PGAS model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. However, PGAS also provides access to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ provides numerous methods for accessing and using global memory. In UPC++, all operations that access remote memory are explicit, which encourages programmers to be aware of the cost of communication and data movement. Moreover, all remote-memory access operations are by default asynchronous, to enable programmers to write code that scales well even on hundreds of thousands of cores.

97 MATHEMATICS AND COMPUTING↗

UPC++ v1.0 Programmer’s Guide, Revision 2020.3.0

UPC++ is a C++11 library that provides Partitioned Global Address Space (PGAS) programming. It is designed for writing parallel programs that run efficiently and scale well on distributed-memory parallel computers. The PGAS model is single program, multiple-data (SPMD), with each separate constituent process having access to local memory as it would in C++. However, PGAS also provides access to a global address space, which is allocated in shared segments that are distributed over the processes. UPC++ provides numerous methods for accessing and using global memory. In UPC++, all operations that access remote memory are explicit, which encourages programmers to be aware of the cost of communication and data movement. Moreover, all remote-memory access operations are by default asynchronous, to enable programmers to write code that scales well even on hundreds of thousands of cores.

97 MATHEMATICS AND COMPUTING↗

UPC++ Programmer’s Guide (v1.0 2017.9)

UPC++ is a C++11 library that provides Asynchronous Partitioned Global Address Space (APGAS) programming. It is designed for writing parallel programs that run efficiently and scale well on distributed-memory parallel computers. The APGAS model is single program, multiple-data (SPMD), with each separate thread of execution (referred to as a rank, a term borrowed from MPI) having access to local memory as it would in C++. However, APGAS also provides access to a global address space, which is allocated in shared segments that are distributed over the ranks. UPC++ provides numerous methods for accessing and using global memory. In UPC++, all operations that access remote memory are explicit, which encourages programmers to be aware of the cost of communication and data movement. Moreover, all remote-memory access operations are by default asynchronous, to enable programmers to write code that scales well even on hundreds of thousands of cores.

97 MATHEMATICS AND COMPUTING↗