Engineering Papers⌕ Search

Engineering topics

Heitmann, Katrin

Publications and source records attributed to Heitmann, Katrin.

At least 55 records · Page 3

High Energy Physics Exascale Requirements Review. An Office of Science review sponsored jointly by Advanced Scientific Computing Research and High Energy Physics, June 10-12, 2015, Bethesda, Maryland

The U.S. Department of Energy (DOE) Office of Science (SC) Offices of High Energy Physics (HEP) and Advanced Scientific Computing Research (ASCR) convened a programmatic Exascale Requirements Review on June 10–12, 2015, in Bethesda, Maryland. This report summarizes the findings, results, and recommendations derived from that meeting. The high-level findings and observations are as follows. Larger, more capable computing and data facilities are needed to support HEP science goals in all three frontiers: Energy, Intensity, and Cosmic. The expected scale of the demand at the 2025 timescale is at least two orders of magnitude — and in some cases greater — than that available currently. The growth rate of data produced by simulations is overwhelming the current ability of both facilities and researchers to store and analyze it. Additional resources and new techniques for data analysis are urgently needed. Data rates and volumes from experimental facilities are also straining the current HEP infrastructure in its ability to store and analyze large and complex data volumes. Appropriately configured leadership-class facilities can play a transformational role in enabling scientific discovery from these datasets. A close integration of high-performance computing (HPC) simulation and data analysis will greatly aid in interpreting the results of HEP experiments. Such an integration will minimize data movement and facilitate interdependent workflows. Long-range planning between HEP and ASCR will be required to meet HEP’s research needs. To best use ASCR HPC resources, the experimental HEP program needs (1) an established, long-term plan for access to ASCR computational and data resources, (2) the ability to map workflows to HPC resources, (3) the ability for ASCR facilities to accommodate workflows run by collaborations potentially comprising thousands of individual members, (4) to transition codes to the next-generation HPC platforms that will be available at ASCR facilities, (5) to build up and train a workforce capable of developing and using simulations and analysis to support HEP scientific research on next-generation systems.

97 MATHEMATICS AND COMPUTING↗

PICS: SIMULATIONS OF STRONG GRAVITATIONAL LENSING IN GALAXY CLUSTERS

Gravitational lensing has become one of the most powerful tools available for investigating the “dark side” of the universe. Cosmological strong gravitational lensing, in particular, probes the properties of the dense cores of dark matter halos over decades in mass and offers the opportunity to study the distant universe at flux levels and spatial resolutions otherwise unavailable. Studies of strongly lensed variable sources offer even further scientific opportunities. One of the challenges in realizing the potential of strong lensing is to understand the statistical context of both the individual systems that receive extensive follow-up study, as well as that of the larger samples of strong lenses that are now emerging from survey efforts. Motivated by these challenges, we have developed an image simulation pipeline, Pipeline for Images of Cosmological Strong lensing (PICS), to generate realistic strong gravitational lensing signals from group- and cluster-scale lenses. PICS uses a low-noise and unbiased density estimator based on (resampled) Delaunay Tessellations to calculate the density field; lensed images are produced by ray-tracing images of actual galaxies from deep Hubble Space Telescope observations. Other galaxies, similarly sampled, are added to fill in the light cone. The pipeline further adds cluster member galaxies and foreground stars into the lensed images. The entire image ensemble is then observed using a realistic point-spread function that includes appropriate detector artifacts for bright stars. Noise is further added, including such non-Gaussian elements as noise window-paning from mosaiced observations, residual bad pixels, and cosmic rays. The aim is to produce simulated images that appear identical—to the eye (expert or otherwise)—to real observations in various imaging surveys.

79 ASTRONOMY AND ASTROPHYSICS↗

Simulations of the pairwise kinematic Sunyaev-Zel'dovich signal

The pairwise kinematic Sunyaev–Zel’dovich (kSZ) signal from galaxy clusters is a probe of their line of sight momenta, and thus a potentially valuable source of cosmological information. In addition to the momenta, the amplitude of the measured signal depends on the properties of the intracluster gas and observational limitations such as errors in determining cluster centers and redshifts. In this work, we simulate the pairwise kSZ signal of clusters at z<1, using the output from a cosmological N-body simulation and including the properties of the intracluster gas via a model that can be varied in post-processing. We find that modifications to the gas profile due to star formation and feedback reduce the pairwise kSZ amplitude of clusters by ∼50%, relative to the naive “gas traces mass” assumption. We demonstrate that miscentering can reduce the overall amplitude of the pairwise kSZ signal by up to 10%, while redshift errors can lead to an almost complete suppression of the signal at small separations. We confirm that a high-significance detection is expected from the combination of data from current generation, high-resolution cosmic microwave background experiments, such as the South Pole Telescope, and cluster samples from optical photometric surveys, such as the Dark Energy Survey. Furthermore, we forecast that future experiments such as Advanced ACTPol in conjunction with data from the Dark Energy Spectroscopic Instrument will yield detection significances of at least 20σ, and up to 57σ in an optimistic scenario. Our simulated maps are publicly available at http://www.hep.anl.gov/cosmology/ksz.html.

79 ASTRONOMY AND ASTROPHYSICS↗

Cosmic Visions Dark Energy: Technology

A strong instrumentation and detector R&D program has enabled the current generation of cosmic frontier surveys. A small investment in R&D will continue to pay dividends and enable new probes to investigate the accelerated expansion of the universe. Instrumentation and detector R&D provide critical training opportunities for future generations of experimentalists, skills that are important across the entire DOE HEP program.

79 ASTRONOMY AND ASTROPHYSICS↗

Efficient construction of mock catalogs for baryon acoustic oscillation surveys

Precision measurements of the large scale structure of the Universe require large numbers of high fidelity mock catalogs to accurately assess, and account for, the presence of systematic effects. We introduce and test a scheme for generating mock catalogs rapidly using suitably derated N-body simulations. Our aim is to reproduce the large scale structure and the gross properties of dark matter halos with high accuracy, while sacrificing the details of the halo's internal structure. By adjusting global and local time-steps in an N-body code, we demonstrate that we recover halo masses to better than 0.5% and the power spectrum to better than 1% both in real and redshift space for k =1 h Mpc{sup −1}, while requiring a factor of 4 less CPU time. We also calibrate the redshift spacing of outputs required to generate simulated light cones. We find that outputs separated by Δ z =0.05 allow us to interpolate particle positions and velocities to reproduce the real and redshift space power spectra to better than 1% (out to k =1 h Mpc{sup −1}). We apply these ideas to generate a suite of simulations spanning a range of cosmologies, motivated by the Baryon Oscillation Spectroscopic Survey (BOSS) but broadly applicable to future large scale structure surveys including eBOSS and DESI. As an initial demonstration of the utility of such simulations, we calibrate the shift in the baryonic acoustic oscillation peak position as a function of galaxy bias with higher precision than has been possible so far. This paper also serves to document the simulations, which we make publicly available.

79 ASTRONOMY AND ASTROPHYSICS↗

Cosmic Visions Dark Energy. Science

Cosmic surveys provide crucial information about high energy physics including strong evidence for dark energy, dark matter, and inflation. Ongoing and upcoming surveys will start to identify the underlying physics of these new phenomena, including tight constraints on the equation of state of dark energy, the viability of modified gravity, the existence of extra light species, the masses of the neutrinos, and the potential of the field that drove inflation. Even after the Stage IV experiments, DESI and LSST, complete their surveys, there will still be much information left in the sky. This additional information will enable us to understand the physics underlying the dark universe at an even deeper level and, in case Stage IV surveys find hints for physics beyond the current Standard Model of Cosmology, to revolutionize our current view of the universe. There are many ideas for how best to supplement and aid DESI and LSST in order to access some of this remaining information and how surveys beyond Stage IV can fully exploit this regime. These ideas flow to potential projects that could start construction in the 2020's.

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS↗

Cosmic Visions Dark Energy: Technology

A strong instrumentation and detector R&D program has enabled the current generation of cosmic frontier surveys. A small investment in R&D will continue to pay dividends and enable new probes to investigate the accelerated expansion of the universe. Instrumentation and detector R&D provide critical training opportunities for future generations of experimentalists, skills that are important across the entire Department of Energy High Energy Physics program.

46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND ↗

THE MIRA–TITAN UNIVERSE: PRECISION PREDICTIONS FOR DARK ENERGY SURVEYS

Large-scale simulations of cosmic structure formation play an important role in interpreting cosmological observations at high precision. The simulations must cover a parameter range beyond the standard six cosmological parameters and need to be run at high mass and force resolution. A key simulation-based task is the generation of accurate theoretical predictions for observables using a finite number of simulation runs, via the method of emulation. Using a new sampling technique, we explore an eight-dimensional parameter space including massive neutrinos and a variable equation of state of dark energy. We construct trial emulators using two surrogate models (the linear power spectrum and an approximate halo mass function). The new sampling method allows us to build precision emulators from just 26 cosmological models and to systematically increase the emulator accuracy by adding new sets of simulations in a prescribed way. Emulator fidelity can now be continuously improved as new observational data sets become available and higher accuracy is required. Finally, using one ΛCDM cosmology as an example, we study the demands imposed on a simulation campaign to achieve the required statistics and accuracy when building emulators for investigations of dark energy.

79 ASTRONOMY AND ASTROPHYSICS↗

Utilizing many-core accelerators for halo and center finding within a cosmology simulation

Efficiently finding and computing statistics about “halos” (regions of high density) are essential analysis steps for N-body cosmology simulations. However, in state-of-the-art simulation codes, these analysis operators do not currently take advantage of the shared-memory data-parallelism available on multi-core and many-core architectures. The Hybrid / Hardware Accelerated Cosmology Code (HACC) is designed as an MPI+X code, but the analysis operators are parallelized only among MPI ranks, because of the difficulty in porting different X implementations (e.g., OpenMP, CUDA) across all architectures on which it is run. In this paper, we present portable data-parallel algorithms for several variations of halo finding and halo center finding algorithms. These are implemented with the PISTON component of the VTK-m framework, which uses Nvidia's Thrust library to construct data-parallel algorithms that allow a single implementation to be compiled to multiple backends to target a variety of multi-core and many-core architectures. Finally, we compare the performance of our halo and center finding algorithms against the original HACC implementations on the Moonlight, Stampede, and Titan supercomputers. The portability of Thrust allowed the same code to run efficiently on each of these architectures. On Titan, the performance improvements using our code have enabled halo analysis to be performed on a very large data set (81923 particles across 16,384 nodes of Titan) for which analysis using only the existing CPU algorithms was not feasible.

Sewell, Christopher↗

COSMIC EMULATION: FAST PREDICTIONS FOR THE GALAXY POWER SPECTRUM

The halo occupation distribution (HOD) approach has proven to be an effective method for modeling galaxy clustering and bias. In this approach, galaxies of a given type are probabilistically assigned to individual halos in N-body simulations. In this paper, we present a fast emulator for predicting the fully nonlinear galaxy–galaxy auto and galaxy–dark matter cross power spectrum and correlation function over a range of freely specifiable HOD modeling parameters. The emulator is constructed using results from 100 HOD models run on a large ΛCDM N-body simulation, with Gaussian Process interpolation applied to a PCA-based representation of the galaxy power spectrum. The total error is currently ∼1% in the auto correlations and ∼2% in the cross correlations from z = 1 to z = 0, over the considered parameter range. We use the emulator to investigate the accuracy of various analytic prescriptions for the galaxy power spectrum, parametric dependencies in the HOD model, and the behavior of galaxy bias as a function of HOD parameters. Additionally, we obtain fully nonlinear predictions for tangential shear correlations induced by galaxy–galaxy lensing from our galaxy–dark matter cross power spectrum emulator. All emulation products are publicly available at http://www.hep.anl.gov/cosmology/CosmicEmu/emu.html.

79 ASTRONOMY AND ASTROPHYSICS↗

The Q continuum simulation: Harnessing the power of GPU accelerated supercomputers

Modeling large-scale sky survey observations is a key driver for the continuing development of high-resolution, large-volume, cosmological simulations. We report the first results from the “Q Continuum” cosmological N-body simulation run carried out on the GPU-accelerated supercomputer Titan. The simulation encompasses a volume of (1300 Mpc){sup 3} and evolves more than half a trillion particles, leading to a particle mass resolution of m{sub p}≃1.5⋅10{sup 8} M{sub ⊙}. At this mass resolution, the Q Continuum run is currently the largest cosmology simulation available. It enables the construction of detailed synthetic sky catalogs, encompassing different modeling methodologies, including semi-analytic modeling and sub-halo abundance matching in a large, cosmological volume. Here we describe the simulation and outputs in detail and present first results for a range of cosmological statistics, such as mass power spectra, halo mass functions, and halo mass-concentration relations for different epochs. We also provide details on challenges connected to running a simulation on almost 90% of Titan, one of the fastest supercomputers in the world, including our usage of Titan’s GPU accelerators.

79 ASTRONOMY AND ASTROPHYSICS↗

HACC: Simulating sky surveys on state-of-the-art supercomputing architectures

Current and future surveys of large-scale cosmic structure are associated with a massive and complex datastream to study, characterize, and ultimately understand the physics behind the two major components of the ‘Dark Universe’, dark energy and dark matter. In addition, the surveys also probe primordial perturbations and carry out fundamental measurements, such as determining the sum of neutrino masses. Large-scale simulations of structure formation in the Universe play a critical role in the interpretation of the data and extraction of the physics of interest. Just as survey instruments continue to grow in size and complexity, so do the supercomputers that enable these simulations. In this paper, we report on HACC (Hardware/Hybrid Accelerated Cosmology Code), a recently developed and evolving cosmology N-body code framework, designed to run efficiently on diverse computing architectures and to scale to millions of cores and beyond. HACC can run on all current supercomputer architectures and supports a variety of programming models and algorithms. It has been demonstrated at scale on Cell- and GPU-accelerated systems, standard multi-core node clusters, and Blue Gene systems. HACC’s design allows for ease of portability, and at the same time, high levels of sustained performance on the fastest supercomputers available. We present a description of the design philosophy of HACC, the underlying algorithms and code structure, and outline implementation details for several specific architectures. We show selected accuracy and performance results from some of the largest high resolution cosmological simulations so far performed, including benchmarks evolving more than 3.6 trillion particles.

79 ASTRONOMY AND ASTROPHYSICS↗

Large-scale compute-intensive analysis via a combined in-situ and co-scheduling workflow approach, In: SC '15 Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Large-scale simulations can produce hundreds of terabytes to petabytes of data, complicating and limiting the efficiency of workflows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in-situ, utilizing the same resources as the simulation, and/or off-loading subsets of the data to a compute-intensive analysis system. We introduce an analysis framework developed for HACC, a cosmological N-body code, that uses both in-situ and co-scheduling approaches for handling petabyte-scale outputs. We compare different analysis set-ups ranging from purely off-line, to purely in-situ to in-situ/co-scheduling. The analysis routines are implemented using the PISTON/VTK-m framework, allowing a single implementation of an algorithm that simultaneously targets a variety of GPU, multi-core, and many-core architectures.

Computer Science↗

Large-Scale Compute-Intensive Analysis via a Combined In-situ and Co-scheduling Workflow Approach

Large-scale simulations can produce tens of terabytes of data per analysis cycle, complicating and limiting the efficiency of workflows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in situ, utilizing the same resources as the simulation, and/or off-loading subsets of the data to a compute-intensive analysis system. We introduce an analysis framework developed for HACC, a cosmological N-body code, that uses both in situ and co-scheduling approaches for handling Petabyte-size outputs. An initial in situ step is used to reduce the amount of data to be analyzed, and to separate out the data-intensive tasks handled off-line. The analysis routines are implemented using the PISTON/VTK-m framework, allowing a single implementation of an algorithm that simultaneously targets a variety of GPU, multi-core, and many-core architectures.

Big data↗

Large-scale structure formation with massive neutrinos and dynamical dark energy

Over the next decade, cosmological measurements of the large-scale structure of the Universe will be sensitive to the combined effects of dynamical dark energy and massive neutrinos. The matter power spectrum is a key repository of this information. We extend higher-order perturbative methods for computing the power spectrum to investigate these effects over quasilinear scales. Through comparison with N-body simulations, we establish the regime of validity of a time-renormalization group perturbative treatment that includes dynamical dark energy and massive neutrinos. We also quantify the accuracy of standard, renormalized and Lagrangian resummation (LPT) perturbation theories without massive neutrinos. We find that an approximation that neglects neutrino clustering as a source for nonlinear matter clustering predicts the baryon acoustic oscillation (BAO) peak position to 0.25% accuracy for redshifts 1 ≤ z ≤ 3, justifying the use of LPT for BAO reconstruction in upcoming surveys. Here, we release a modified version of the public Copter code which includes the additional physics discussed in the paper.

79 ASTRONOMY AND ASTROPHYSICS↗