Data for "NSTX-U research advancing the physics of spherical tokamaks"
Explore the source record for details and available documents.
Engineering topics
Publications and source records attributed to Dominski, Julien.
Explore the source record for details and available documents.
We present the Exascale Framework for High Fidelity coupled Simulations (EFFIS), a workflow and code coupling framework developed as part of the Whole Device Modeling Application (WDMApp) in the Exascale Computing Project. EFFIS consists of a library, command line utilities, and a collection of run-time daemons. Together, these software products enable users to easily compose and execute workflows that include: strong or weak coupling, in situ (or offline) analysis/visualization/monitoring, command-and-control actions, remote dashboard integration, and more. We describe WDMApp physics coupling cases and computer science requirements that motivate the design of the EFFIS framework. Furthermore, we explain the essential enabling technology that EFFIS leverages: ADIOS for performant data movement, PerfStubs/TAU for performance monitoring, and an advanced COUPLER for transforming coupling data from its native format to the representation needed by another application. Finally, we demonstrate EFFIS using coupled multi-simulation WDMApp workflows and exemplify how the framework supports the project’s needs. We show that EFFIS and its associated services for data movement, visualization, and performance collection does not introduce appreciable overhead to the WDMApp workflow and that the resource-dominant application’s idle time while waiting for data is minimal.
In this paper, we present a scheme that spatially couples two gyrokinetic codes using first-principles. Coupled equations are presented and a necessary and sufficient condition for ensuring accuracy is derived. This new scheme couples both the field and the particle distribution function. The coupling of the distribution function is only performed once every few time-steps, using a five-dimensional (5D) grid to communicate the distribution function between the two codes. This 5D grid interface enables the coupling of different types of codes and models, such as particle and continuum codes, or delta-f and total-f models. Transferring information from the 5D grid to the marker particle weights is achieved using a new resampling technique. Demonstration of the coupling scheme is shown using two XGC gyrokinetic simulations for both the core and edge. We also apply the coupling scheme to two continuum simulations for a one-dimensional advection–diffusion problem.
Two existing particle-in-cell gyrokinetic codes, GEM for the core region and XGC for the edge region, have been successfully coupled with a spatial coupling scheme at the interface in a toroidal geometry. Additionally, a mapping technique is developed for transferring data between GEM's structured and XGC's unstructured meshes. Two examples of coupled simulations are presented to demonstrate the coupling scheme. The optimization of GEM for graphics processing unit is also presented.
Self-interaction is the process by which a microinstability eigenmode that is extended along the direction parallel to the magnetic field interacts non-linearly with itself. This effect is particularly significant in gyrokinetic simulations accounting for kinetic passing electron dynamics and is known to generate stationary $E\times B$ zonal flow shear layers at radial locations near low-order mode rational surfaces (Weikl et al. Phys. Plasmas, vol. 25, 2018, 072305). Here, we find that self-interaction, in fact, plays a very significant role in also generating fluctuating zonal flows, which is critical to regulating turbulent transport throughout the radial extent. Unlike the usual picture of zonal flow drive in which microinstability eigenmodes coherently amplify the flow via modulational instabilities, the self-interaction drive of zonal flows from these eigenmodes are uncorrelated with each other. It is shown that the associated shearing rate of the fluctuating zonal flows therefore reduces as more toroidal modes are resolved in the simulation. In simulations accounting for the full toroidal domain, such an increase in the density of toroidal modes corresponds to an increase in the toroidal system size, leading to a finite system size effect that is distinct from the well-known profile shearing effect.
We present ContourNet, a deep learning approach to identify salient local isocontours as blobs in large-scale 5D gyrokinetic tokamak simulation data. Blobs-regions of high turbulence that run along the edge wall down toward the diverter and can damage the tokamak-are non-well-defined features but have been empirically localized by isocontours in 2D normalized fluctuating density fields. The key of our study is to train ContourNet to follow the empirical rules to detect blobs over the time-varying simulation data. The architecture of ContourNet is a convolutional neural segmentation network: the inputs are the density field and a rasterized isocontour; the output is a set of isocontour encircling blobs. At the training stage, we feed the network with manually identified isocontours and propagated labels. At the inference stage, we extract isocontours from the segmented blob regions. Results show that our approach can achieve both high accuracy and performance, which enables scientists to understand the blob dynamics influencing the confinement of the plasma.
Complex workflows consisting of multiple simulation and analysis codes running concurrently through in-memory coupling is becoming popular due to inherent advantages in online management of large-scale data, resilience, and the code development process. However, orchestrating such a multi-application workflow to efficiently utilize resources on a heterogeneous architecture is challenging. In this paper, we present our results with running the Fusion Whole Device Modeling benchmark workflow on Summit, a pre-exascale supercomputer at Oak Ridge National Laboratory. We explore various resource distribution and process placement mechanisms, including sharing compute nodes between processes from separate applications. We show that fine-grained process placement can have a significant impact towards efficient utilization of the compute power of a node on Summit, and conclude that sophisticated tools for performing co-design studies of multi-application workflows can play an important role towards efficient orchestration of such workflows.
Part of the promise of exascale computing and the next generation of scientific simulation codes is the ability to bring together time and spatial scales that have traditionally been treated separately. This enables creating complex coupled simulations and in situ analysis pipelines, encompassing such things as "whole device" fusion models or the simulation of cities from sewers to rooftops. Unfortunately, the HPC analysis tools that have been built up over the preceding decades are ill suited to the debugging and performance analysis of such computational ensembles. In this paper, we present a new vision for performance measurement and understanding of HPC codes, MonitoringAnalytics (MONA). MONA is designed to be a flexible, high performance monitoring infrastructure that can perform monitoring analysis in place or in transit by embedding analytics and characterization directly into the data stream, without relying upon delivering all monitoring information to a central database for post-processing. It addresses the trade-offs between the prohibitively expensive capture of all performance characteristics and not capturing enough to detect the features of interest. We demonstrate several uses of MONA; capturing and indexing multi-executable performance profiles to enable later processing, extraction of performance primitives to enable the generation of customizable benchmarks and performance skeletons, and extracting communication and application behaviors to enable better control and placement for the current and future runs of the science ensemble. Relevant performance information based on a system for MONA built from ADIOS and SOSflow technologies is provided for DOE science applications and leadership machines.
With the growing computational complexity of science and the complexity of new and emerging hardware, it is time to re-evaluate the traditional monolithic design of computational codes. One new paradigm is constructing larger scientific computational experiments from the coupling of multiple individual scientific applications, each targeting their own physics, characteristic lengths, and/or scales. We present a framework constructed by leveraging capabilities such as in-memory communications, workflow scheduling on HPC resources, and continuous performance monitoring. This code coupling capability is demonstrated by a fusion science scenario, where differences between the plasma at the edges and at the core of a device have different physical descriptions. This infrastructure not only enables the coupling of the physics components, but it also connects in situ or online analysis, compression, and visualization that accelerate the time between a run and the analysis of the science content. Results from runs on Titan and Cori are presented as a demonstration.
A new scheme that tightly couples kinetic turbulence codes across a spatial interface is introduced. This scheme evolves from considerations of competing strategies and down-selection. It is found that the use of a composite kinetic distribution function and fields with global boundary conditions as if the coupled code were one makes the coupling problem tractable. In contrast, coupling the two solutions from each code across the overlap region is found to be more difficult due to numerical dephasing of the turbulent solutions between two solvers. Finally, another advantage of the new scheme is that the data movement can be limited to the 3D fluid quantities, instead of higher dimensional kinetic information, which is computationally more efficient for large scale simulations on leadership class computers.
The trends in high performance computing, where far more data can be computed that can ever be stored, have made in situ techniques an important area of research and development. Simulation campaigns, where domain scientists work with computer scientists to run a simulation and perform in situ analysis and visualization are important, and complex undertakings. In this paper we report our experiences performing in situ analysis and visualization on two campaigns. The two campaigns were related, but had important differences in terms of the codes that were used, the types of analysis and visualization required, and the visualization tools used. Further, we report the lessons learned from each campaign.
A new adaptive scheme to be used in particle-in-cell codes for carrying out gyroaveraging operations with matrices is presented. This new scheme uses an intermediate velocity grid whose resolution is adapted to the local thermal Larmor radius. The charge density is computed by projecting marker weights in a field-line following manner while preserving the adiabatic magnetic moment μ. These choices permit to improve the accuracy of the gyroaveraging operations performed with matrices even when strong spatial variation of temperature and magnetic field is present. Accuracy of the scheme in different geometries from simple 2D slab geometry to realistic 3D toroidal equilibrium has been studied. As a result, a successful implementation in the gyrokinetic code XGC is presented in the delta-f limit.
Axisymmetric (n=0) density fluctuations measured in the TCV tokamak are observed to possess a frequency f<sub>0</sub> which is either varying (radially dispersive oscillations) or a constant over a large fraction of the plasma minor radius (radially global oscillations) as reported in a companion paper [Z. Huang <i>et al.</i>, this issue]. Given that f<sub>0</sub> scales with the sound speed and given the poloidal structure of density fluctuations, these oscillations were interpreted as Geodesic Acoustic Modes, even though f<sub>0</sub> is in fact smaller than the local linear GAM frequency f<sub>GAM</sub> . In this work we employ the Eulerian gyrokinetic code GENE to simulate TCV relevant conditions and investigate the nature properties of these oscillations, in particular their relation to the safety factor profile. Local and global simulations are carried out and a good qualitative agreement is observed between experiments and simulations. By varying also the plasma temperature and density profiles, we conclude that a variation of the edge safety factor alone is not sufficient to induce a transition from global to radially inhomogeneous oscillations, as was initially suggested by experimental results. This transition appears instead to be the combined result of variations in the different plasma profiles, collisionality and finite machine size effects. In conclusion, simulations also show that radially global GAM-like oscillations can be observed in all fluxes and fluctuation fields, suggesting that they are the result of a complex nonlinear process involving also finite toroidal mode numbers and not just linear global GAM eigenmodes.