Engineering topics

Frontiere, Nicholas

Publications and source records attributed to Frontiere, Nicholas.

At least 19 records

DOE OSTI2024

Advances in ArborX to support exascale applications

ArborX is a performance portable geometric search library developed as part of the Exascale Computing Project (ECP). In this paper, we explore a collaboration between ArborX and a cosmological simulation code HACC. Large cosmological simulations on exascale platforms encounter a bottleneck due to the in-situ analysis requirements of halo finding, a problem of identifying dense clusters of dark matter (halos). This problem is solved by using a density-based DBSCAN clustering algorithm. With each MPI rank handling hundreds of millions of particles, it is imperative for the DBSCAN implementation to be efficient. In addition, the requirement to support exascale supercomputers from different vendors necessitates performance portability of the algorithm. We describe how this challenge problem guided ArborX development, and enhanced the performance and the scope of the library. We explore the improvements in the basic algorithms for the underlying search index to improve the performance, and describe several implementations of DBSCAN in ArborX. Further, we report the history of the changes in ArborX and their effect on the time to solve a representative benchmark problem, as well as demonstrate the real world impact on production end-to-end cosmology simulations.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2024

A Cast of Thousands: How the IDEAS Productivity Project Has Advanced Software Productivity and Sustainability

Computational and data-enabled science and engineering are revolutionizing advances throughout science and society, at all scales of computing. For example, teams in the U.S. Department of Energy’s Exascale Computing Project have been tackling new frontiers in modeling, simulation, and analysis by exploiting unprecedented exascale computing capabilities—building an advanced software ecosystem that supports next-generation applications and addresses disruptive changes in computer architectures. However, concerns are growing about the productivity of the developers of scientific software. Members of the Interoperable Design of Extreme-scale Application Software project serve as catalysts to address these challenges through fostering software communities, incubating and curating methodologies and resources, and disseminating knowledge to advance developer productivity and software sustainability. This article discusses how these synergistic activities are advancing scientific discovery—mitigating technical risks by building a firmer foundation for reproducible, sustainable science at all scales of computing, from laptops to clusters to exascale and beyond.

97 MATHEMATICS AND COMPUTING↗

DOE OSTI2023

Experiences Readying Applications for Exascale

The advent of Exascale computing invites an assessment of existing best practices for developing application readiness on the world's largest supercomputers. This work details observations from the last four years in preparing scientific applications to run on the Oak Ridge Leadership Computing Facility's (OLCF) Frontier system. This paper addresses a range of topics in software including programmability, tuning, and portability considerations that are key to moving applications from existing systems to future installations. A set of representative workloads provides case studies for general system and software testing. We evaluate the use of early access systems for development across several generations of hardware. Finally, we discuss how best practices were identified and disseminated to the community through a wide range of activities including user-guides and trainings. We conclude with recommendations for ensuring application readiness on future leadership computing systems.

exascale↗

DOE OSTI2023

Frontier: Exploring Exascale

As the US Department of Energy (DOE) computing facilities began deploying petascale systems in 2008, DOE was already setting its sights on exascale. In that year, DARPA published a report on the feasibility of reaching exascale. The report authors identified several key challenges in the pursuit of exascale including power, memory, concurrency, and resiliency. That report informed the DOE's computing strategy for reaching exascale. With the deployment of Oak Ridge National Laboratory's Frontier supercomputer, we have officially entered the exascale era. In this paper, we discuss Frontier's architecture, how it addresses those challenges, and describe some early application results from Oak Ridge Leadership Computing Facility's Center of Excellence and the Exascale Computing Project.

Atchley, Scott {Leadership Computing}↗

DOE OSTI2023

Experiences readying applications for Exascale

Gottiparthi, Kalyan↗

DOE OSTI2023

Improving initialization and evolution accuracy of cosmological neutrino simulations

Neutrino mass constraints are a primary focus of current and future large-scale structure (LSS) surveys. Non-linear LSS models rely heavily on cosmological simulations — the impact of massive neutrinos should therefore be included in these simulations in a realistic, computationally tractable, and controlled manner. Here, a recent proposal to reduce the related computational cost employs a symmetric neutrino momentum sampling strategy in the initial conditions. We implement a modified version of this strategy into the Hardware/Hybrid Accelerated Cosmology Code (HACC) and perform convergence tests on its internal parameters. We illustrate that this method can impart Script $\mathcal{O}$(1%) numerical artifacts on the total matter field on small scales, similar to previous findings, and present a method to remove these artifacts using Fourier-space filtering of the neutrino density field. Moreover, we show that the converged neutrino power spectrum does not follow linear theory predictions on relatively large scales at early times at the 15% level, prompting a more careful study of systematics in particle-based neutrino simulations. We also present an improved method for backscaling linear transfer functions for initial conditions in massive neutrino cosmologies that is based on achieving the same relative neutrino growth as computed with Boltzmann solvers. Our self-consistent backscaling method yields sub-percent accuracy in the total matter growth function. Comparisons for the non-linear power spectrum with the Mira-Titan emulator at a neutrino mass of m ν = 0.15 eV are in very good agreement with the expected level of errors in the emulator and in the direct N-body simulation.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2023

Numerical discreteness errors in multispecies cosmological N -body simulations

ABSTRACT We present a detailed analysis of numerical discreteness errors in two-species, gravity-only, cosmological simulations using the density power spectrum as a diagnostic probe. In a simple set-up where both species are initialized with the same total matter transfer function, biased growth of power forms on small scales when the solver force resolution is finer than the mean interparticle separation. The artificial bias is more severe when individual density and velocity transfer functions are applied. In particular, significant large-scale offsets in power are measured between simulations with conventional offset grid initial conditions when compared against converged high-resolution results where the force resolution scale is matched to the interparticle separation. These offsets persist even when the cosmology is chosen so that the two particle species have the same mass, indicating that the error is sourced from discreteness in the total matter field as opposed to unequal particle mass. We further investigate two mitigation strategies to address discreteness errors: the frozen potential method and softened interspecies short-range forces. The former evolves particles under the approximately ‘frozen’ total matter potential in linear theory at early times, while the latter filters cross-species gravitational interactions on small scales in low-density regions. By modelling closer to the continuum limit, both mitigation strategies demonstrate considerable reductions in large-scale power spectrum offsets.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2023

Simulating Hydrodynamics in Cosmology with CRK-HACC

Abstract We introduce CRK-HACC, an extension of the Hardware/Hybrid Accelerated Cosmology Code (HACC), to resolve gas hydrodynamics in large-scale structure formation simulations of the universe. The new framework couples the HACC gravitational N -body solver with a modern smoothed-particle hydrodynamics (SPH) approach called conservative reproducing kernel SPH (CRKSPH). CRKSPH utilizes smoothing functions that exactly interpolate linear fields while manifestly preserving conservation laws (momentum, mass, and energy). The CRKSPH method has been incorporated to accurately model baryonic effects in cosmology simulations—an important addition targeting the generation of precise synthetic sky predictions for upcoming observational surveys. CRK-HACC inherits the codesign strategies of the HACC solver and is built to run on modern GPU-accelerated supercomputers. In this work, we summarize the primary solver components and present a number of standard validation tests to demonstrate code accuracy, including idealized hydrodynamic and cosmological setups, as well as self-similarity measurements.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

Modelling the Lyman-α forest with Eulerian and SPH hydrodynamical methods

ABSTRACT We compare two state-of-the-art numerical codes to study the overall accuracy in modelling the intergalactic medium and reproducing Lyman-α forest observables for DESI and high-resolution data sets. The codes employ different approaches to solving both gravity and modelling the gas hydrodynamics. The first code, Nyx, solves the Poisson equation using the Particle-Mesh (PM) method and the Euler equations using a finite-volume method. The second code, CRK-HACC , uses a Tree-PM method to solve for gravity, and an improved Lagrangian smoothed particle hydrodynamics (SPH) technique, where fluid elements are modelled with particles, to treat the intergalactic gas. We compare the convergence behaviour of the codes in flux statistics as well as the degree to which the codes agree in the converged limit. We find good agreement overall with differences being less than observational uncertainties, and a particularly notable ≲1 per cent agreement in the 1D flux power spectrum. This agreement was achieved by applying a tessellation methodology for reconstructing the density in CRK-HACC instead of using an SPH kernel as is standard practice. We show that use of the SPH kernel can lead to significant and unnecessary biases in flux statistics; this is especially prominent at high redshifts, z ∼ 5, as the Lyman-α forest mostly comes from lower-density regions that are intrinsically poorly sampled by SPH particles.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2022

Farpoint: A High-resolution Cosmology Simulation at the Gigaparsec Scale

Abstract In this paper we introduce the Farpoint simulation, the latest member of the Hardware/Hybrid Accelerated Cosmology Code (HACC) gravity-only simulation family. The domain covers a volume of (1000 h −1 Mpc) 3 and evolves close to two trillion particles, corresponding to a mass resolution of m p ∼ 4.6 × 10 7 h −1 M ⊙ . These specifications enable comprehensive investigations of the galaxy–halo connection, capturing halos down to small masses. Further, the large volume resolves scales typical of modern surveys with good statistical coverage of high-mass halos. The simulation was carried out on the GPU-accelerated system Summit, one of the fastest supercomputers currently available. We provide specifics about the Farpoint run and present an initial set of results. The high mass resolution facilitates precise measurements of important global statistics, such as the halo concentration–mass relation and the correlation function down to small scales. Selected subsets of the simulation data products are publicly available via the HACC Simulation Data Portal.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

The Last Journey. II. SMACC—Subhalo Mass-loss Analysis Using Core Catalogs

Here we introduce Subhalo Mass-loss Analysis using Core Catalogs (SMACC). SMACC adds a mass model to substructure merger trees based on halo "core tracking." Our approach avoids the need for running expensive subhalo finding algorithms and instead uses subhalo mass-loss modeling to assign masses to halo cores. We present details of the SMACC methodology and demonstrate its excellent performance in describing halo substructure and its evolution. Validation of the approach is carried out using cosmological simulations at significantly different resolutions. We apply SMACC to the 1.24-trillion-particle Last Journey simulation and construct core catalogs with the additional mass information. These catalogs can be readily used as input to semianalytic models or subhalo abundance matching approaches to determine approximate galaxy distributions, as well as for in-depth studies of small-scale structure evolution.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2021

The Last Journey. I. An Extreme-scale Simulation on the Mira Supercomputer

The Last Journey is a large-volume, gravity-only, cosmological N-body simulation evolving more than 1.24 trillion particles in a periodic box with a side length of 5.025 Gpc. It was implemented using the HACC simulation and analysis framework on the BG/Q system Mira. The cosmological parameters are chosen to be consistent with the results from the Planck satellite. A range of analysis tools have been run in situ to enable a diverse set of science projects and, at the same time, keep the resulting data amount manageable. Analysis outputs have been generated starting at redshift z ~ 10 to allow for construction of synthetic galaxy catalogs using a semianalytic modeling approach in postprocessing. As part of our in situ analysis pipeline, we employ a new method for tracking halo substructures, introducing the concept of subhalo cores. The production of multiwavelength synthetic sky maps is facilitated by generating particle light cones in situ, also beginning at z ~ 10. We provide an overview of the simulation setup and generated data products; a first set of analysis results is presented. A subset of the data is publicly available.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2020

The Mira-Titan Universe. III. Emulation of the Halo Mass Function

We construct an emulator for the halo mass function over group and cluster mass scales for a range of cosmologies, including the effects of dynamical dark energy and massive neutrinos. The emulator is based on the recently completed Mira-Titan Universe suite of cosmological N-body simulations. The main set of simulations spans 111 cosmological models with 2.1 Gpc boxes. We extract halo catalogs in the redshift range z = [0.0, 2.0] and for masses M-200c >= 10(13)M(circle dot)/h. The emulator covers an eight-dimensional hypercube spanned by {Omega(m)h(2), Omega(b)h(2), Omega(nu)h(2), sigma(8), h, n(s), w(0), w(a)}; spatial flatness is assumed. We obtain smooth halo mass functions by fitting piecewise second-order polynomials to the halo catalogs and employ Gaussian process regression to construct the emulator while keeping track of the statistical noise in the input halo catalogs and uncertainties in the regression process. For redshifts z less than or similar to 1, the typical emulator precision is better than 2% for 10(13)-10(14)M(circle dot)/h and <10<^> M similar or equal to 101(circle dot)(54)/h. For comparison, fitting functions using the traditional universal form for the halo mass function can be biased at up to 30% at M similar or equal to 10(144)M(circle dot)/h for z = 0. Our emulator is publicly available at https://github.com/SebastianBocquet/MiraTitanHMFemulator.

cosmology: theory↗

DOE OSTI2019

CosmoDC2: A Synthetic Sky Catalog for Dark Energy Science with LSST

This paper introduces cosmoDC2, a large synthetic galaxy catalog designed to support precision dark energy science with the Large Synoptic Survey Telescope (LSST). CosmoDC2 is the starting point for the second data challenge (DC2) carried out by the LSST Dark Energy Science Collaboration (LSST DESC). The catalog is based on a trillion-particle, (4.225 Gpc) 3 box cosmological N-body simulation, the Outer Rim run. It covers 440 deg 2 of sky area to a redshift of z = 3 and matches expected number densities from contemporary surveys to a magnitude depth of 28 in the r band. Each galaxy is characterized by a multitude of galaxy properties including stellar mass, morphology, spectral energy distributions, broadband filter magnitudes, host halo information, and weak lensing shear. The size and complexity of cosmoDC2 requires an efficient catalog generation methodology; our approach is based on a new hybrid technique that combines data-based empirical approaches with semianalytic galaxy modeling. A wide range of observation-based validation tests has been implemented to ensure that cosmoDC2 enables the science goals of the planned LSST DESC DC2 analyses. This paper also represents the official release of the cosmoDC2 data set, including an efficient reader that facilitates interaction with the data.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2019

The Outer Rim Simulation: A Path to Many-core Supercomputers

Herein we describe the Outer Rim cosmological simulation, one of the largest high-resolution N-body simulations performed to date, aimed at promoting science to be carried out with large-scale structure surveys. The simulation covers a volume of (4.225 Gpc) 3 and evolves more than one trillion particles. It was executed on Mira, a BlueGene/Q system at the Argonne Leadership Computing Facility. We discuss some of the computational challenges posed by a system such as Mira, a many-core supercomputer, and how the simulation code, Hardware/Hybrid Accelerated Cosmology Code, has been designed to overcome these challenges. We have carried out a large range of analyses on the simulation data and we report on the results as well as the data products that have been generated. The full data set generated by the simulation totals more than 5 PB of data, making curation and handling of the data a large challenge in itself. The simulation results have been used to generate synthetic catalogs for large-scale structure surveys, including Dark Energy Spectroscopic Instrument and eBOSS, as well as experiments on the cosmic microwave background. A detailed catalog for the data challenges of the Large Synoptic Survey Telescope Dark Energy Science Collaboration has been created as well. We publicly release some of the Outer Rim halo catalogs, downsampled particle information, and lightcone data.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2019

The Borg Cube Simulation: Cosmological Hydrodynamics with CRK-SPH

A challenging requirement posed by next-generation observations is a firm theoretical grasp of the impact of baryons on structure formation. Cosmological hydrodynamic simulations modeling gas physics are vital in this regard. A high degree of modeling flexibility exists in this space, making it important to explore a range of methods in order to gauge the accuracy of simulation predictions. We present results from the first cosmological simulation using Conservative Reproducing Kernel Smoothed Particle Hydrodynamics (CRK-SPH). We employ two simulations: one evolved purely under gravity, and the other with nonradiative hydrodynamics. Each contains 2 × 2304 3 cold dark matter plus baryon particles in an 800 h -1 Mpc box. We compare statistics to previous nonradiative simulations including power spectra, mass functions, baryon fractions, and concentration. We find self-similar radial profiles of gas temperature, entropy, and pressure and show that a simple analytic model recovers these results to better than 40% over two orders of magnitude in mass. We quantify the level of nonthermal pressure support in halos and demonstrate that hydrostatic mass estimates are biased low by 24% (10%) for halos of mass 10 15 (10 13 ) h -1 M⊙. Here, we compute angular power spectra for the thermal and kinematic Sunyaev–Zel’dovich effects and find good agreement with the low-ℓ Planck measurements. Finally, artificial scattering between particles of unequal mass is shown to have a large impact on the gravity-only run, and we highlight the importance of better understanding this issue in hydrodynamic applications. This is the first in a simulation campaign using CRK-SPH, with future work including subresolution gas treatments.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2018

Halo Profiles and the Concentration–Mass Relation for a ΛCDM Universe

Profiles of dark matter-dominated halos at the group and cluster scales play an important role in modern cosmology. Using results from two very large cosmological N-body simulations, which increase the available volume at their mass resolution by roughly two orders of magnitude, we robustly determine the halo concentration-mass (c-M) relation over a wide range of masses, employing multiple methods of concentration measurement. We then characterize individual halo profiles, as well as stacked profiles, relevant for galaxy-galaxy lensing and next-generation cluster surveys; the redshift range covered is 0 ≤ z ≤ 4, with a minimum halo mass of M 200c ~ 2 x 10 11 M ⊙ . Despite the complexity of a proper description of a halo (environmental effects, merger history, nonsphericity, relaxation state), when the mass is scaled by the nonlinear mass scale M * (z), we find that a simple non-power-law form for the c-M/M * relation provides an excellent description of our simulation results across eight decades in M/M * and for 0 ≤ z ≤ 4. Over the mass range covered, the c-M relation has two asymptotic forms: an approximate power law below a mass threshold M/M * ~500-1000, transitioning to a constant value, c 0 ~ 3 at higher masses. The relaxed halo fraction decreases with mass, transitioning to a constant value of ~0.5 above the same mass threshold. We compare Navarro-Frenk-White (NFW) and Einasto fits to stacked profiles in narrow mass bins at different redshifts; as expected, the Einasto profile provides a better description of the simulation results. At cluster scales at low redshift, however, both NFW and Einasto profiles are in very good agreement with the simulation results, consistent with recent weak lensing observations.

79 ASTRONOMY AND ASTROPHYSICS↗

DOE OSTI2017

Building Halo Merger Trees from the Q Continuum Simulation

Cosmological N-body simulations rank among the most computationally intensive efforts today. A key challenge is the analysis of structure, substructure, and the merger history for many billions of compact particle clusters, called halos. Effectively representing the merging history of halos is essential for many galaxy formation models used to generate synthetic sky catalogs, an important application of modern cosmological simulations. Generating realistic mock catalogs requires computing the halo formation history from simulations with large volumes and billions of halos over many time steps, taking hundreds of terabytes of analysis data. We present fast parallel algorithms for producing halo merger trees and tracking halo substructure from a single-level, density-based clustering algorithm. Merger trees are created from analyzing the halo-particle membership function in adjacent snapshots, and substructure is identified by tracking the "cores" of merging halos - sets of particles near the halo center. Core tracking is performed after creating merger trees and uses the relationships found during tree construction to associate substructures with hosts. The algorithms are implemented with MPI and evaluated on a Cray XK7 supercomputer using up to 16,384 processes on data from HACC, a modern cosmological simulation framework. We present results for creating merger trees from 101 analysis snapshots taken from the Q Continuum, a large volume, high mass resolution, cosmological simulation evolving half a trillion particles.

79 ASTRONOMY AND ASTROPHYSICS↗