Engineering Papers⌕ Search

Engineering topics

Bertoni, Colleen

Publications and source records attributed to Bertoni, Colleen.

The Effective Fragment Molecular Orbital Method: Achieving High Scalability and Accuracy for Large Systems

The effective fragment molecular orbital (EFMO) method has been developed to predict the total energy of a very large molecular system accurately (with respect to the underlying quantum mechanical method) and efficiently by taking advantage of the locality of strong chemical interactions and employing a two-level hierarchical parallelism. The accuracy of the EFMO method is partly attributed to the accurate and robust intermolecular interaction prediction between distant fragments, in particular, the many-body polarization and dispersion effects, which require the generation of static and dynamic polarizability tensors by solving the coupled perturbed Hartree–Fock (CPHF) and time-dependent HF (TDHF) equations, respectively. Solving the CPHF and TDHF equations is the main EFMO computational bottleneck due to the inefficient (serial) and I/O-intensive implementation of the CPHF and TDHF solvers. In this work, the efficiency and scalability of the EFMO method are significantly improved with a new CPU memory-based implementation for solving the CPHF and TDHF equations that are parallelized by either message passing interface (MPI) or hybrid MPI/OpenMP. Here, the accuracy of the EFMO method is demonstrated for both covalently bonded systems and noncovalently bound molecular clusters by systematically examining the effects of basis sets and a key distance-related cutoff parameter, R cut . R cut determines whether a fragment pair (dimer) is treated by the chosen ab initio method or calculated using the effective fragment potential (EFP) method (separated dimers). Decreasing the value of Rcut increases the number of separated (EFP) dimers, thereby decreasing the computational effort. It is demonstrated that excellent accuracy (<1 kcal/mol error per fragment) can be achieved when using a sufficiently large basis set with diffuse functions coupled with a small R cut value. With the new parallel implementation, the total EFMO wall time is substantially reduced, especially with a high number of MPI ranks. Given a sufficient workload, nearly ideal strong scaling is achieved for the CPHF and TDHF parts of the calculation. For the first time, EFMO calculations with the inclusion of long-range polarization and dispersion interactions on a hydrated mesoporous silica nanoparticle with explicit water solvent molecules (more than 15k atoms) are achieved on a massively parallel supercomputer using nearly 1000 physical nodes. In addition, EFMO calculations on the carbinolamine formation step of an amine-catalyzed aldol reaction at the nanoscale with explicit solvent effects are presented.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

The General Atomic and Molecular Electronic Structure System (GAMESS): Novel Methods on Novel Architectures

The primary focus of GAMESS over the last 5 years has been the development of new high-performance codes that are able to take effective and efficient advantage of the most advanced computer architectures, both CPU and accelerators. These efforts include employing density fitting and fragmentation methods to reduce the high scaling of well-correlated (e.g., coupled-cluster) methods as well as developing novel codes that can take optimal advantage of graphical processing units and other modern accelerators. Because accurate wave functions can be very complex, an important new functionality in GAMESS is the quasi-atomic orbital analysis, an unbiased approach to the understanding of covalent bonds embedded in the wave function. Finally, best practices for the maintenance and distribution of GAMESS are also discussed.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

HIPLZ: Enabling performance portability for exascale systems

While heterogeneous computing has emerged as a dominant trend in current and future High-Performance Computing (HPC) systems, it is also widely recognized that this shift has led to increased software complexity due to a proliferation of programming systems for different heterogeneous processors. One such example is the Heterogeneous-Compute Interface for Portability from AMD (HIP ), which is composed of a C Runtime API and C++ Kernel Language. Many HPC applications will likely use HIP on future exascale systems (e.g., Frontier and El Capitan), but HIP currently only targets AMD and NVIDIA processors. This limitation creates challenges for users who would also like to run their applications on exascale systems based on other architectures (e.g., Aurora, which is based on Intel hardware) that are currently not targeted by HIP . In this paper, we introduce the design and implementation of HIPLZ , a compiler and runtime system that uses the Intel Level Zero API to support HIP on Intel GPU architectures. We discuss the design of HIPLZ , derived from HIPCL (an implementation of HIP on top of OpenCL ), and portability issues that occur from using the Level Zero runtime as a backend. We evaluate our implementation by running several performance benchmarks and mini-apps written in HIP on Intel architectures using HIPLZ . Our results show that this approach provides competitive performance relative to Intel's OpenCL implementations on Intel Gen9 and UHD Graphics 770 GPUs, while providing good coverage of features needed by HPC applications. Overall, this approach is a promising demonstration of enabling performance portability for exascale systems.

97 MATHEMATICS AND COMPUTING↗

Porting fragmentation methods to GPUs using an OpenMP API: Offloading the resolution-of-the-identity second-order Møller–Plesset perturbation method

Here, using an OpenMP Application Programming Interface, the resolution-of-the-identity second-order Møller–Plesset perturbation (RI-MP2) method has been off-loaded onto graphical processing units (GPUs), both as a standalone method in the GAMESS electronic structure program and as an electron correlation energy component in the effective fragment molecular orbital (EFMO) framework. First, a new scheme has been proposed to maximize data digestion on GPUs that subsequently linearizes data transfer from central processing units (CPUs) to GPUs. Second, the GAMESS Fortran code has been interfaced with GPU numerical libraries (e.g., NVIDIA cuBLAS and cuSOLVER) for efficient matrix operations (e.g., matrix multiplication, matrix decomposition, and matrix inversion). The standalone GPU RI-MP2 code shows an increasing speedup of up to 7.5× using one NVIDIA V100 GPU with one IBM 42-core P9 CPU for calculations on fullerenes of increasing size from 40 to 260 carbon atoms using the 6-31G(d)/cc-pVDZ-RI basis sets. A single Summit node with six V100s can compute the RI-MP2 correlation energy of a cluster of 175 water molecules using the correlation consistent basis sets cc-pVDZ/cc-pVDZ-RI containing 4375 atomic orbitals and 14 700 auxiliary basis functions in ~0.85 h. In the EFMO framework, the GPU RI-MP2 component shows near linear scaling for a large number of V100s when computing the energy of an 1800-atom mesoporous silica nanoparticle in a bath of 4000 water molecules. The parallel efficiencies of the GPU RI-MP2 component with 2304 and 4608 V100s are 98.0% and 96.1%, respectively.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Prediction of correlation energies using variational subspace valence bond

In the variational subspace valence bond (VSVB) [G. D. Fletcher, J. Chem. Phys. 142, 134112 (2015)] method, the electronic orbitals comprising the wave function correspond to chemically meaningful objects, such as bonds, lone pairs, atomic cores, and so on. Selected regions of a molecule (for example, a single chemical bond, as opposed to all the bonds) can be modeled with different levels of basis set and possible methods for modeling correlation from the other regions. The interactions between the components of a molecule (say, a bond and a neighboring orbital) can then be studied in detail for their impact on a chemical phenomenon while avoiding the expense of necessarily applying the higher levels and methods to the entire molecule. Here, this work presents the theoretical basis for modeling correlation effects between specific electron pairs by incorporating terms in the inter-electronic coordinates (“r 12 ”) into VSVB. The approach is validated with calculations on small systems using single-reference wave functions.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

OpenMP application experiences: Porting to accelerated nodes

As recent enhancements to the OpenMP specification become available in its implementations, there is a need to share the results of experimentation in order to better understand the OpenMP implementation’s behavior in practice, to identify pitfalls, and to learn how the implementations can be effectively deployed in scientific codes. We report on experiences gained and practices adopted when using OpenMP to port a variety of ECP applications, mini-apps and libraries based on different computational motifs to accelerator-based leadership-class high-performance supercomputer systems at the United States Department of Energy. Additionally, we identify important challenges and open problems related to the deployment of OpenMP. Through our report of experiences, we find that OpenMP implementations are successful on current supercomputing platforms and that OpenMP is a promising programming model to use for applications to be run on emerging and future platforms with accelerated nodes.

97 MATHEMATICS AND COMPUTING↗

Outcomes of OpenMP Hackathon: OpenMP Application Experiences with the Offloading Model (Part II)

This paper reports on experiences gained and practices adopted when using the latest features of OpenMP to port a variety of HPC applications and mini-apps based on different computational motifs (BerkeleyGW, WDMApp/XGC, GAMESS, GESTS, and GridMini) to accelerator-based, leadership-class, high-performance supercomputer systems at the Department of Energy. As recent enhancements to OpenMP become available in implementations, there is a need to share the results of experimentation with them in order to better understand their behavior in practice, to identify pitfalls, and to learn how they can be effectively deployed in scientific codes. Additionally, we identify best practices from these experiences that we can share with the rest of the OpenMP community.

Chapman, Barbara↗

Outcomes of OpenMP Hackathon: OpenMP Application Experiences with the Offloading Model (Part I)

This paper reports on experiences gained and practices adopted when using the latest features of OpenMP to port a variety of HPC applications and mini-apps based on different computational motifs (BerkeleyGW, WDMApp/XGC, GAMESS, GESTS, and GridMini) to accelerator-based, leadership-class, high-performance supercomputer systems at the Department of Energy. As recent enhancements to OpenMP become available in implementations, there is a need to share the results of experimentation with them in order to better understand their behavior in practice, to identify pitfalls, and to learn how they can be effectively deployed in scientific codes. Additionally, we identify best practices from these experiences that we can share with the rest of the OpenMP community.

Chapman, Barbara↗

Recent Developments in the General Atomic and Molecular Electronic Structure System

A discussion of many of the recently implemented features of GAMESS (General Atomic and Molecular Electronic Structure System) and LibCChem (the C++ CPU/GPU library associated with GAMESS) is presented. These features include fragmentation methods like the fragment molecular orbital, effective fragment potential and effective fragment molecular orbital methods, hybrid MPI/OpenMP approaches to Hartree-Fock and resolution of the identity second order perturbation theory. Many new coupled cluster theory methods have been implemented in GAMESS, as have multiple levels of density functional/tight binding theory. The role of accelerators, especially graphical processing units, is discussed in the context of the new features of LibCChem, as is the associated problem of power consumption as the power of computers increases dramatically. The process by which a complex program suite like GAMESS is maintained and developed is considered. Future developments are briefly summarized.

74 ATOMIC AND MOLECULAR PHYSICS↗

Performance Portability Evaluation of OpenCL Benchmarks across Intel and NVIDIA Platforms

We evaluate the capabilities of vendor-provided OpenCL implementations for performance portability across multiple computing platforms. The Rodinia benchmark suite is used for this evaluation. We apply the metric defined by Pennycook et al., and we use roofline efficiency from the Roofline performance model as the "performance efficiency" in the metric's definition. We found that the delivered performance portability is similar for several benchmarks, even if the roofline-based performance efficiencies across platforms are very different among the benchmarks. To help distinguish between these instances, we extend the metric by adding the standard deviation of the performance efficiencies for each benchmark. We argue that the standard deviation gives additional insight into performance portability assessment since it adds the performance variability across platforms. Additionally, we discuss the challenges to measure performance portability associated with algorithms and system software. In terms of algorithms, we need to carefully construct the benchmarks and appropriately use the concurrency available on a platform. In terms of system software, we depend on the vendor performance tools to support the desired programming model and runtime to be able to measure the metrics of interest.

GPU↗

Valence : A Massively Parallel Implementation of the Variational Subspace Valence Bond Method

This work describes the software package, Valence, for the calculation of molecular energies using the variational subspace valence bond (VSVB) method. VSVB is an ab initio electronic structure method based on nonorthogonal orbitals. Important features of practical value include high parallel scalability, wave functions that can be constructed automatically by combining orbitals from previous calculations, and ground and excited states that can be modeled with a single configuration or determinant. Here, the open-source software package includes tools to generate wave functions, a database of generic orbitals, example input files, and a library build intended for integration with other packages. We also present the interface to an external software package, enabling the computation of optimized molecular geometries and vibrational frequencies.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Developing a Computational Chemistry Framework for the Exascale Era

Within computational chemistry the NWChem package has arguably been the de facto standard for running high-accuracy numerical simulations on the most powerful supercomputers. In order to better address the challenges presented by emerging exascale architectures, the decision has been made to rewrite NWChem. Design of the resulting package, NWChemEx, has been driven by exascale computing; however, significant additional design considerations have arisen from the team's involvement with the Molecular Sciences Software Institute (MolSSI). MolSSI is a National Science Foundation initiative focused on establishing coding and data standards for the computational chemistry community. As a result, NWChemEx is built upon a general computational chemistry framework called the Simulation Development Environment (SDE) that is designed with a focus on extensibility and interoperability. The present manuscript describes the modular approach of the SDE and how it has been used to implement the self-consistent field algorithm within NWChemEx.

97 MATHEMATICS AND COMPUTING↗

RAM as a Network Managed Resource

This paper describes an architecture for and several application examples of using a dynamically allocable RAM pool over the network as part of a deep memory hierarchy. We present four different use cases, including in situ analysis, machine learning, quantum chemistry simulations, and virtualization. In each use case, we modified or implemented software to use a network RAM pool, and then evaluated the performance. The cases of in situ analysis, machine learning, and virtualization demonstrated good performance. While the quantum chemistry experiments exhibited an average slowdown of 53% for all runs, including those where resources were clearly oversubscribed, the scalability results showed promise.

Allcock, William↗

Perspective: Ab initio force field methods derived from quantum mechanics

It is often desirable to accurately and efficiently model the behavior of large molecular systems in the condensed phase (thousands to tens of thousands of atoms) over long time scales (from nanoseconds to milliseconds). In these cases, ab initio methods are difficult due to the increasing computational cost with the number of electrons. A more computationally attractive alternative is to perform the simulations at the atomic level using a parameterized function to model the electronic energy. Many empirical force fields have been developed for this purpose. However, the functions that are used to model interatomic and intermolecular interactions contain many fitted parameters obtained from selected model systems, and such classical force fields cannot properly simulate important electronic effects. Furthermore, while such force fields are computationally affordable, they are not reliable when applied to systems that differ significantly from those used in their parameterization. They also cannot provide the information necessary to analyze the interactions that occur in the system, making the systematic improvement of the functional forms that are used difficult. Ab initio force field methods aim to combine the merits of both types of methods. The ideal ab initio force fields are built on first principles and require no fitted parameters. Ab initio force field methods surveyed in this perspective are based on fragmentation approaches and intermolecular perturbation theory. This perspective summarizes their theoretical foundation, key components in their formulation, and discusses key aspects of these methods such as accuracy and formal computational cost. The ab initio force fields considered here were developed for different targets, and this perspective also aims to provide a balanced presentation of their strengths and shortcomings. Finally, this perspective suggests some future directions for this actively developing area.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗

Multipole moments in the effective fragment potential method

In the effective fragment potential (EFP) method the Coulomb potential is represented using a set of multipole moments generated by the distributed multipole analysis (DMA) method. Misquitta, Stone, and Fazeli recently developed a basis space-iterated stockholder atom (BS-ISA) method to generate multipole moments. This study assesses the accuracy of the EFP interaction energies using sets of multipole moments generated from the BS-ISA method, and from several versions of the DMA method (such as analytic and numeric grid-based), with varying basis sets. Both methods lead to reasonable results, although using certain implementations of the DMA method can result in large errors. With respect to the CCSD(T)/CBS interaction energies, the mean unsigned error (MUE) of the EFP method for the S22 data set using BS-ISA–generated multipole moments and DMA-generated multipole moments (using a small basis set and the analytic DMA procedure) is 0.78 and 0.72 kcal/mol, respectively. Here, the MUE accuracy is on the same order as MP2 and SCS-MP2. The MUEs are lower than in a previous study benchmarking the EFP method without the EFP charge transfer term, demonstrating that the charge transfer term increases the accuracy of the EFP method. Regardless of the multipole moment method used, it is likely that much of the error is due to an insufficient short-range electrostatic term (i.e., charge penetration term), as shown by comparisons with symmetry-adapted perturbation theory.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗

Using quantum chemistry muscle to flex massive systems: How to respond to something perturbing

Computational chemistry uses the theoretical advances of quantum mechanics and the algorithmic and hardware advances of computer science to give insight into chemical problems. It is currently possible to do highly accurate quantum chemistry calculations, but the most accurate methods are very computationally expensive. Thus it is only feasible to do highly accurate calculations on small molecules, since typically more computationally efficient methods are also less accurate. The overall goal of my dissertation work has been to try to decrease the computational expense of calculations without decreasing the accuracy. In particular, my dissertation work focuses on fragmentation methods, intermolecular interactions methods, analytic gradients, and taking advantage of new hardware.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗