Using C++ AMP to accelerate HPC applications on Multiple Platforms
Explore the source record for details and available documents.
Engineering topics
Publications and source records attributed to Elwasif, Wael R.
Explore the source record for details and available documents.
Batteries are highly complex electrochemical systems, with performance and safety governed by coupled nonlinear electrochemical-electrical-thermal-mechanical processes over a range of spatiotemporal scales. We describe a new, open source computational environment for battery simulation known as VIBE - the Virtual Integrated Battery Environment. VIBE includes homogenized and pseudo-2D electrochemistry models such as those by Newman-Tiedemann-Gu (NTG) and Doyle- Fuller-Newman (DFN, a.k.a. DualFoil) as well as a new advanced capability known as AMPERES (Advanced MultiPhysics for Electrochemical and Renewable Energy Storage). AMPERES provides a 3D model for electrochemistry and full coupling with 3D electrical and thermal models on the same grid. VIBE/AMPERES has been used to create three-dimensional battery cell and pack models that explicitly simulate all the battery components (current collectors, electrodes, and separator). The models are used to predict battery performance under normal operations and to study thermal and mechanical response under adverse conditions.
Explore the source record for details and available documents.
High-performance computing (HPC) workloads are increasingly leveraging loosely coupled large scale simula- tions. Unfortunately, most large-scale HPC platforms, including Cray/ALPS environments, are designed for the execution of long-running jobs based on coarse-grained launch capabilities (e.g., one MPI rank per core on all allocated compute nodes). This assumption limits capability-class workload campaigns that require large numbers of discrete or loosely coupled simulations, and where time-to-solution is an untenable pacing issue. This paper describes the challenges related to the support of fine-grained launch capabilities that are necessary for the execution of loosely coupled large scale simulations on Cray/ALPS platforms. More precisely, we present the details of an enhanced runtime system to support this use case, and report on initial results from early testing on systems at Oak Ridge National Laboratory.
This work demonstrates that simulations of advanced burning plasma operation scenarios can be successfully parallelized in time using the parareal algorithm. CORSICA - an advanced operation scenario code for tokamak plasmas is used as a test case. This is a unique application since the parareal algorithm has so far been applied to relatively much simpler systems except for the case of turbulence. In the present application, a computational gain of an order of magnitude has been achieved which is extremely promising. A successful implementation of the Parareal algorithm to codes like CORSICA ushers in the possibility of time efficient simulations of ITER plasmas.
Explore the source record for details and available documents.
Parareal is an iterative algorithm that, in effect, achieves temporal decomposition for a time-dependent system of differential or partial differential equations. A solution is obtained in a shorter wall-clock time, but at the expense of increased compute cycles. The algorithm combines a fine solver that solves the system to acceptable accuracy with an approximate coarse solver. The critical task for the successful implementation of parareal on any system is the development of a coarse solver that leads to convergence in a small number of iterations compared to the number of time slices in the full time interval, and is, at the same time, much faster than the fine solver. Fast coarse solvers may not lead to sufficiently rapid convergence, and slow coarse solvers may not lead to significant gains even if the number of iterations to convergence is satisfactory. We find that the difficulty of meeting these conflicting demands can be substantially eased by using a data-driven, event-based implementation of parareal instead of the conventional algorithm where solver tasks are executed sequentially. For given convergence properties, the event-based approach relaxes the speed requirements on the coarse solver by a factor of , where is the number of iterations required for a converged solution. This may, for many problems, lead to an efficient parareal implementation that would otherwise not be possible or would require substantial coarse solver development.
Parareal is a recent algorithm able to parallelize the time dimension in spite of its sequential nature. It has been applied to several linear and nonlinear problems and, very recently, to a simulation of fully-developed, two-dimensional drift wave turbulence. The mere fact that parareal works in such a turbulent regime is in itself somewhat unexpected, due to the characteristic sensitivity of turbulence to any change in initial conditions. This fundamental property of any turbulent system should render the iterative correction procedure characteristic of the parareal method inoperative, but this seems not to be the case. In addition, the choices that must be made to implement parareal (division of the temporal domain, election of the coarse solver and so on) are currently made using trial-and-error approaches. Here, we identify the mechanisms responsible for the convergence of parareal of these simulations of drift wave turbulence. We also investigate which conditions these mechanisms impose on any successful parareal implementation. The results reported here should be useful to guide future implementations of parareal within the much wider context of fully-developed fluid and plasma turbulent simulations.
The increasing availability of large scale computing capabilities has accelerated the development of high-fidelity coupled simulations. Such simulations typically involve the integration of models that implement various aspects of the complex phenomena under investigation. Coupled simulations are playing an integral role in fields such as climate modeling, earth systems modeling, rocket simulations, computational chemistry, fusion research, and many other computational fields. Model coupling provides scientists with systematic ways to virtually explore the physical, mathematical, and computational aspects of the problem. Such exploration is rarely done using a single execution of a simulation, but rather by aggregating the results from many simulation runs that, together, serve to bring to light novel knowledge about the system under investigation. Furthermore, it is often the case (particularly in engineering disciplines) that the study of the underlying system takes the form of an optimization regime, where the control parameter space is explored to optimize an objective functions that captures system realizability, cost, performance, or a combination thereof. Novel and flexible frameworks that facilitate the integration of the disparate models into a holistic simulation are used to perform this research, while making efficient use of the available computational resources. In this paper, we describe the integration of the DAKOTA optimization and parameter sweep toolkit with the Integrated Plasma Simulator (IPS), a component-based framework for loosely coupled simulations. The integration allows DAKOTA to exploit the internal task and resource management of the IPS to dynamically instantiate simulation instances within a single IPS instance, allowing for greater control over the trade-off between efficiency of resource utilization and time to completion. We present a case study showing the use of the combined DAKOTA-IPS system to aid in the design of a lithium ion battery (LIB) cell, by studying a coupled system involving the electrochemistry and ion transport at the lower length scales and thermal energy transport at the device scales. The DAKOTA-IPS system provides a flexible tool for use in optimization and parameter sweep studies involving loosely coupled simulations that is suitable for use in situations where changes to the constituent components in the coupled simulation are impractical due to intellectual property or code heritage issues.
This paper discusses on-going work with the Integrated Plasma Simulator (IPS), a framework for coupled multiphysics simulations of plasmas, to allow simulations to run through the loss of nodes on which the simulation is executing. While many different techniques are available to improve the fault tolerance of computational science applications on high-performance computer systems, checkpoint/restart (C/R) remains virtually the only one that see widespread use in practice. Our focus here is to augment the traditional C/R approach with additional techniques that can provide a more localized and tailored response to faults based on the ability to restart failed tasks on an individual basis, and the use of information external to the application itself in order to guide decision-making, in many cases avoiding the need to stop and restart the entire simulation. This capability involves several features within the IPS framework, and leverages the Fault Tolerance Backplane, a publish/subscribe event service to disseminate fault-related information throughout HPC systems, to obtain information from the Reliability, Availability and Serviceability (RAS) subsystem of the HPC system. This work is described in the context of Cray XT-series computer systems for concreteness, but is applicable to other environments as well. As part of the analysis of this work, we discuss the requirements to generalize this approach to other complex simulation applications beyond the Integrated Plasma Simulator.
Parareal is a novel algorithm that allows the solution of time-dependent systems of differential or partial differential equations (PDE) to be parallelized in the temporal domain. Parareal-based implementations of PDE problems can take advantage of this parallelism to significantly reduce the time to solution for a simulation (though at an increased total cost) while making effective use of the much larger processor counts available on current high-end systems. In this paper, we present a dynamic, dependency-driven version of the parareal algorithm which breaks the final sequential bottleneck remaining in the original formulation, making it amenable to a "many-task" treatment. We further improve the cost and execution time of the algorithm by introducing a moving window for time slices, which avoids the execution of tasks which contribute little to the final global solution. We describe how this approach has been realized in the Integrated Plasma Simulator (IPS), a framework for coupled multiphysics simulations, and examine the trade-offs among time-to-solution, total cost, and resource utilization efficiency as a function of the compute resources applied to the problem.
No abstract prepared.
Simulation of fusion plasmas involve a broad range of timescales. In magnetically confined plasmas, such as in ITER, the timescale associated with the microturbulence responsible for transport and confinement timescales vary by an order of 10^6 10^9. Simulating this entire range of timescales is currently impossible, even on the most powerful supercomputers available. Space parallelization has so far been the most common approach to solve partial differential equations. Space parallelization alone has led to computational saturation for fluid codes, which means that the walltime for computaion does not linearly decrease with the increasing number of processors used. The application of the parareal algorithm to simulations of fusion plasmas ushers in a new avenue of parallelization, namely temporal parallelization. The algorithm has been successfully applied to plasma turbulence simulations, prior to which it has been applied to other relatively simpler problems. This work explores the extension of the applicability of the parareal algorithm to ITER relevant problems, starting with a diffusion-convection model.
This paper discusses the Integrated Plasma Simulator (IPS), a framework for coupled multiphysics simulation of fusion plasmas, in the context of many-task computing. The IPS supports multiple levels of parallelism: individual computational tasks can be parallel, components can launch multiple tasks concurrently, tasks from multiple components can be executed concurrently within a simulation, and multiple simulations can be run simultaneously. Each level of parallelism is constructed on top of the many-task computing capabilities implemented in the IPS, the foundation for the parallelism presented at the multiple simulation level. We show that a modest number of simultaneous simulations, with appropriately sized resource allocations, can provide a better trade-off between resource utilization and overall execution time than if they are run as separate jobs. This approach is highly beneficial for situations in which individual simulation tasks may differ significantly in parallel scalability, as is the case in many scientific communities where coupled simulations rely substantially on legacy code.
As computing capabilities have increased, the coupling of computational models has become an increasingly viable and therefore important way of improving the physical fidelity of simulations. Applications currently using some form of multicode or multi-component coupling include climate modeling, rocket simulations, and chemistry. In recent years, the plasma physics community has also begun to pursue integrated multiphysics simulations for space weather and fusion energy applications. Such model coupling generally exposes new issues in the physical, mathematical, and computational aspects of the problem. This paper focuses on the computational aspects of one such effort, detailing the design, and implementation of the Integrated Plasma Simulator (IPS) for the Center for Simulation of Wave Interactions with Magnetohydrodynamics (SWIM). The IPS framework focuses on maximizing flexibility for the creators of loosely-coupled component-based simulations, and provides services for execution coordination, resource management, data management, and inter-component communication. It also serves as a proving ground for a concurrent 'multi-tasking' execution model to improve resource utilization, and application-level fault tolerance. We also briefly describe how the IPS has been applied to several problems of interest to the fusion community.
The Integrated Plasma Simulator (IPS) provides a framework within which some of the most advanced, massively-parallel fusion modeling codes can be interoperated to provide a detailed picture of the multi-physics processes involved in fusion experiments. The presentation will cover four topics: (1) recent improvements to the IPS, (2) application of the IPS for very high resolution simulations of ITER scenarios, (3) studies of resistive and ideal MHD stability in tokamak discharges using IPS facilities, and (4) the application of RF power in the electron cyclotron range of frequencies to control slowly growing MHD modes in tokamaks and initial evaluations of optimized location for RF power deposition.
Concerns over the environment and energy security have recently prompted renewed interest in the U.S. in nuclear energy. Recognizing this, the U.S. Dept. of Energy has launched an initiative to revamp and modernize the role that modeling and simulation plays in the development and operation of nuclear facilities. This Nuclear Energy Advanced Modeling and Simulation (NEAMS) program represents a major investment in the development of new software, with one or more large multi-scale multi-physics capabilities in each of four technical areas associated with the nuclear fuel cycle, as well as additional supporting developments. In conjunction with this, we are designing a software architecture, computational environment, and component framework to integrate the NEAMS technical capabilities and make them more accessible to users. In this report of work very much in progress, we lay out the 'problem' we are addressing, describe the model-driven system design approach we are using, and compare them with several large-scale technical software initiatives from the past. We discuss how component technology may be uniquely positioned to address the software integration challenges of the NEAMS program, outline the capabilities planned for the NEAMS computational environment and framework, and describe some initial prototyping activities.
In the component approach, components are said to interact through well-defined interfaces. Conventionally, these interfaces are based on method signatures which define the syntax, if not the semantics of the interface. However in many cases, this does not encompass all of the interactions a component has with its peers. In the Integrated Plasma Simulator (IPS), a framework for coupled simulation of fusion plasmas developed by the Center for Simulation of RF Wave Interactions with Magnetohydrodynamics (SWIM), we find that components interact through files and through an event service in addition to the usual method calls. In this report, we discuss the motivations for the design of the IPS and our experience with it. We further consider whether the concept of a component interface needs to be extended, and how it might be done. Component-based software engineering (CBSE) has allowed applications to grow in size, scale and power, by coupling components along well-defined interfaces. Many groups have come together to provide rules and standards for component and interface definitions. General purpose component architectures have been designed for business, and scientific computing, while domain specific component frameworks have been created in many domains, including climate modeling, rocket simulation, and engineering. In each architecture, component interfaces, typically implemented as method invocations, are the only ways that are mentioned for components to interact with each other and the outside world. However, components may not always communicate over their interfaces. They can interact with software entities in many other ways including sharing files and publish/subscribe systems. In this paper we consider two ways in which components interact without method invocation in a component framework, how the component interface can be extended to describe these interactions, and the implications on the greater component community. The rest of the paper continues as follows: first we briefly describe the fusion simulation framework that serves as the basis for the discussion (Sec. 2), followed by a description of non-interface interactions (Sec. 3), a discussion of how to extend to the component interface (Sec. 4) and finally some conclusions (Sec. 5).