Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “ensemble simulations”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 289 records · Page 16
All sourcesNASA NTRSDOE OSTI
DOE OSTI2024

RASPA3

RASPA3, a molecular simulation code for computing adsorption and diffusion in nanoporous materials and thermodynamic and transport properties of fluids. It implements force field based classical Monte Carlo/molecular dynamics in various ensembles. RASPA3 is rewritten from the ground up in C++23 with speed and code readability in mind. Transition-matrix Monte Carlo is added to compute the density of states and free energies. The Monte Carlo code for rigid molecules is based on quaternions, and the atomic positions needed in the energy evaluation are recreated from the center of mass position and quaternion orientation. The expanded ensemble methodology for fractional molecules, with a scaling parameter λ between 0 and 1, now also keeps track of analytic expressions of dU/dλ, allowing independent verification of the chemical potential using thermodynamic integration. The source code is freely available under the MIT license on GitHub.

Dubbeldam, David↗
DOE OSTI2023

Pyrocumulonimbus Events Over British Columbia in 2017: An Ensemble Model Study of Parameter Sensitivities and Climate Impacts of Wildfire Smoke in the Stratosphere

Abstract The Pyrocumulonimbus (pyroCb) events over British Columbia in 2017 were observed in the lower stratosphere for about 8–10 months after the smoke injections. Several previous studies used global climate models to investigate the physical parameters for the 2017 pyroCb events, but the conclusions show strong model dependency. In this study, we use the Energy Exascale Earth System Model (E3SM) and complete an ensemble of runs exploring three injection parameters: smoke aerosol mass, the percentage of black carbon within the smoke aerosols, and plume injection height. Additionally, we consider the heterogeneous reaction of ozone and primary organic matter. According to the satellite daily observed aerosol optical depth (AOD), we find that the best ensemble member is the simulation with 0.4 Tg of smoke, 3% of which is black carbon, a 13.5 km smoke injection height, and a 10 −5 probability factor of the heterogeneous reaction. Besides AOD, we examine the ensemble score based on the metrics used in previous studies: the metrics of the extinction coefficient at 18 km altitude and the maximum plume height. The conclusion of the best estimate of the injection parameters for 2017 pyroCb events shows strong not only model but also evaluation metric dependency. We use the Random Forest machine learning technique to quantify the relative importance of each parameter in accurately simulating the 2017 pyroCb events and find that the injection height is the most critical feature, no matter which metric is used to score the ensemble members.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Huge ensembles – Part 1: Design of ensemble weather forecasts using spherical Fourier neural operators

Abstract. Simulating low-likelihood high-impact extreme weather events in a warming world is a significant and challenging task for current ensemble forecasting systems. While these systems presently use up to 100 members, larger ensembles could enrich the sampling of internal variability. They may capture the long tails associated with climate hazards better than traditional ensemble sizes. Due to computational constraints, it is infeasible to generate huge ensembles (comprised of 1000–10 000 members) with traditional, physics-based numerical models. In this two-part paper, we replace traditional numerical simulations with machine learning (ML) to generate hindcasts of huge ensembles. In Part 1, we construct an ensemble weather forecasting system based on spherical Fourier neural operators (SFNOs), and we discuss important design decisions for constructing such an ensemble. The ensemble represents model uncertainty through perturbed-parameter techniques, and it represents initial condition uncertainty through bred vectors, which sample the fastest-growing modes of the forecast. Using the European Centre for Medium-Range Weather Forecasts Integrated Forecasting System (IFS) as a baseline, we develop an evaluation pipeline composed of mean, spectral, and extreme diagnostics. With large-scale, distributed SFNOs with 1.1 billion learned parameters, we achieve calibrated probabilistic forecasts. As the trajectories of the individual members diverge, the ML ensemble mean spectra degrade with lead time, consistent with physical expectations. However, the individual ensemble members' spectra stay constant with lead time. Therefore, these members simulate realistic weather states during the rollout, and the ML ensemble passes a crucial spectral test in the literature. The IFS and ML ensembles have similar extreme forecast indices, and we show that the ML extreme weather forecasts are reliable and discriminating. These diagnostics ensure that the ensemble can reliably simulate the time evolution of the atmosphere, including low-likelihood high-impact extremes. In Part 2, we generate a huge ensemble initialized each day in summer 2023, and we characterize the simulations of extremes.

Mahesh, Ankur↗
DOE OSTI2020

On the Correspondence between Seasonal Forecast Biases and Long-Term Climate Biases in Sea Surface Temperature

In this paper, the correspondence between mean sea surface temperature (SST) biases in retrospective seasonal forecasts (hindcasts) and long-term climate simulations from five global climate models is examined to diagnose the degree to which systematic SST biases develop on seasonal time scales. The hindcasts are from the North American Multi-Model Ensemble and the climate simulations are from the Coupled Model Intercomparison Project. The analysis suggests that most robust climatological SST biases begin to form within 6 months of a realistically initialized integration, although the growth rate varies with location, time, and model. In regions with large biases, interannual variability and ensemble spread is much smaller than the climatological bias. Additional ensemble hindcasts of the Community Earth System Model with a different initialization method suggest that initial conditions do matter for the initial bias growth, but the overall global bias patterns are similar after 6 months. A hindcast approach is more suitable to study biases over the tropics and sub-tropics than over the extra-tropics because of smaller initial biases and faster bias growth. The rapid emergence of SST biases makes it likely that fast processes with times scales shorter than the seasonal time scales in the atmosphere and upper ocean are responsible for a substantial part of the climatological SST biases. Studying the growth of biases may provide important clues to the causes and ultimately the amelioration of these biases. Further, initialized seasonal hindcasts can profitably be used in the development of high-resolution coupled ocean-atmosphere models.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Finite temperature auxiliary field quantum Monte Carlo in the canonical ensemble

Finite temperature auxiliary field-based quantum Monte Carlo methods, including determinant quantum Monte Carlo and Auxiliary Field Quantum Monte Carlo (AFQMC), have historically assumed pivotal roles in the investigation of the finite temperature phase diagrams of a wide variety of multidimensional lattice models and materials. Despite their utility, however, these techniques are typically formulated in the grand canonical ensemble, which makes them difficult to apply to condensates such as superfluids and difficult to benchmark against alternative methods that are formulated in the canonical ensemble. Working in the grand canonical ensemble is furthermore accompanied by the increased overhead associated with having to determine the chemical potentials that produce desired fillings. Given this backdrop, in this work, we present a new recursive approach for performing AFQMC simulations in the canonical ensemble that does not require knowledge of chemical potentials. To derive this approach, we exploit the convenient fact that AFQMC solves the many-body problem by decoupling many-body propagators into integrals over one-body problems to which non-interacting theories can be applied. We benchmark the accuracy of our technique on illustrative Bose and Fermi–Hubbard models and demonstrate that it can converge more quickly to the ground state than grand canonical AFQMC simulations. We believe that our novel use of HS-transformed operators to implement algorithms originally derived for non-interacting systems will motivate the development of a variety of other methods and anticipate that our technique will enable direct performance comparisons against other many-body approaches formulated in the canonical ensemble.

Shen, Tong (ORCID:0000000206292468)↗
DOE OSTI2022

Enabling machine learning-ready HPC ensembles with Merlin

With the growing complexity of computational and experimental facilities, many scientific researchers are turning to machine learning (ML) techniques to analyze large scale ensemble data. With complexities such as multi-component workflows, heterogeneous machine architectures, parallel file systems, and batch scheduling, care must be taken to facilitate this analysis in a high performance computing (HPC) environment. Here, we present Merlin, a workflow framework to enable large ML-friendly ensembles of scientific HPC simulations. By augmenting traditional HPC with distributed compute technologies, Merlin aims to lower the barrier for scientific subject matter experts to incorporate ML into their analysis. As a producer–consumer workflow model, Merlin enables multi-machine, cross-batch job, dynamically allocated yet persistent workflows capable of utilizing surge-compute resources. Key features of Merlin are a flexible HPC-centric interface, low per-task overhead, multi-tiered fault recovery, and a hierarchical sampling algorithm that allows for $\mathscr{O}$(N) task execution and $\mathscr{O}$(N ln N) task queuing to ensembles of millions of tasks. In addition to Merlin’s design, we test the algorithm’s performance in an HPC center and demonstrate the ability to enqueue 40 million simulations in 100 s, with a 30 millisecond per-task overhead that is independent of ensemble size. Finally, we describe some example applications that Merlin has enabled on leadership-class HPC resources, such as the ML-augmented optimization of nuclear fusion experiments and the calibration of infectious disease models to study the progression of and possible mitigation strategies for COVID-19.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2020

Protein folding from heterogeneous unfolded state revealed by time-resolved X-ray solution scattering

One of the most challenging tasks in biological science is to understand how a protein folds. In theoretical studies, the hypothesis adopting a funnel-like free-energy landscape has been recognized as a prominent scheme for explaining protein folding in views of both internal energy and conformational heterogeneity of a protein. Despite numerous experimental efforts, however, comprehensively studying protein folding with respect to its global conformational changes in conjunction with the heterogeneity has been elusive. Here we investigate the redox-coupled folding dynamics of equine heart cytochrome c (cyt-c) induced by external electron injection by using time-resolved X-ray solution scattering. A systematic kinetic analysis unveils a kinetic model for its folding with a stretched exponential behavior during the transition toward the folded state. With the aid of the ensemble optimization method combined with molecular dynamics simulations, we found that during the folding the heterogeneously populated ensemble of the unfolded state is converted to a narrowly populated ensemble of folded conformations. These observations obtained from the kinetic and the structural analyses of X-ray scattering data reveal that the folding dynamics of cyt-c accompanies many parallel pathways associated with the heterogeneously populated ensemble of unfolded conformations, resulting in the stretched exponential kinetics at room temperature. This finding provides direct evidence with a view to microscopic protein conformations that the cyt-c folding initiates from a highly heterogeneous unfolded state, passes through still diverse intermediate structures, and reaches structural homogeneity by arriving at the folded state.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2020

Scalable molecular dynamics on CPU and GPU architectures with NAMD

NAMD is a molecular dynamics program designed for high-performance simulations of very large biological objects on CPU- and GPU-based architectures. NAMD offers scalable performance on petascale parallel supercomputers consisting of hundreds of thousands of cores, as well as on inexpensive commodity clusters commonly found in academic environments. It is written in C++ and leans on Charm++ parallel objects for optimal performance on low-latency architectures. NAMD is a versatile, multipurpose code that gathers state-of-the-art algorithms to carry out simulations in apt thermodynamic ensembles, using the widely popular CHARMM, AMBER, OPLS, and GROMOS biomolecular force fields. Here, we review the main features of NAMD that allow both equilibrium and enhanced-sampling molecular dynamics simulations with numerical efficiency. We describe the underlying concepts utilized by NAMD and their implementation, most notably for handling long-range electrostatics; controlling the temperature, pressure, and pH; applying external potentials on tailored grids; leveraging massively parallel resources in multiple-copy simulations; and hybrid quantum-mechanical/molecular-mechanical descriptions. We detail the variety of options offered by NAMD for enhanced-sampling simulations aimed at determining free-energy differences of either alchemical or geometrical transformations and outline their applicability to specific problems. Last, we discuss the roadmap for the development of NAMD and our current efforts toward achieving optimal performance on GPU-based architectures, for pushing back the limitations that have prevented biologically realistic billion-atom objects to be fruitfully simulated, and for making large-scale simulations less expensive and easier to set up, run, and analyze. NAMD is distributed free of charge with its source code at www.ks.uiuc.edu.

high-performance computing↗
DOE OSTI2023

Fluctuating hydrodynamics and the Rayleigh–Plateau instability

The Rayleigh–Plateau instability occurs when surface tension makes a fluid column become unstable to small perturbations. At nanometer scales, thermal fluctuations are comparable to interfacial energy densities. Consequently, at these scales, thermal fluctuations play a significant role in the dynamics of the instability. These microscopic effects have previously been investigated numerically using particle-based simulations, such as molecular dynamics (MD), and stochastic partial differential equation–based hydrodynamic models, such as stochastic lubrication theory. In this paper, we present an incompressible fluctuating hydrodynamics model with a diffuse-interface formulation for binary fluid mixtures designed for the study of stochastic interfacial phenomena. An efficient numerical algorithm is outlined and validated in numerical simulations of stable equilibrium interfaces. We present results from simulations of the Rayleigh–Plateau instability for long cylinders pinching into droplets for Ohnesorge numbers of Oh = 0.5 and 5.0. Both stochastic and perturbed deterministic simulations are analyzed and ensemble results show significant differences in the temporal evolution of the minimum radius near pinching. Short cylinders, with lengths less than their circumference, were also investigated. As previously observed in MD simulations, we find that thermal fluctuations cause these to pinch in cases where a perturbed cylinder would be stable deterministically. Finally, we show that the fluctuating hydrodynamics model can be applied to study a broader range of surface tension–driven phenomena.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2023

Large-scale molecular dynamics simulations of bubble collapse in water: Effects of system size, water model, and nitrogen

Molecular dynamics simulations in the microcanonical ensemble are performed to study the collapse of a bubble in liquid water using the single-site mW and the four-site TIP4P/2005 water models. To study system size effects, simulations for pure water systems are performed using periodically replicated simulation boxes with linear dimensions, L, ranging from 32 to 512 nm with the largest systems containing 8.7 × 10 6 and 4.5 × 10 9 molecules for the TIP4P/2005 and mW water models, respectively. The computationally more efficient mW water model allows us to reach converging behavior when the bubble dynamics results are plotted in reduced units, and the limiting behavior can be obtained through linear extrapolation in L –1 . Qualitative differences are observed between simulations with the mW and TIP4P/2005 water models, but they can be explained by the models’ differences in predicted viscosity and surface tension. Although bubble collapse occurs on time scales of only hundreds of picoseconds, the system sizes used here are sufficiently large to obtain bubble dynamics consistent with the Rayleigh–Plesset equation when using the models’ thermophysical properties as input. For the conditions explored here, extreme heating of the interfacial water molecules near the time of collapse is observed for the larger mW water systems (but the model underpredicts the viscosity), whereas heating is less pronounced for the TIP4P/2005 water systems because its larger viscosity contribution slows the collapse dynamics. The presence of nitrogen within the bubble only starts to affect bubble dynamics near the very end of the initial collapse, leading to an incomplete collapse and strong rebound for the mW water model. Although nitrogen is non-condensable at 300 K, it becomes highly compressed and reaches a liquid-like density near the collapse point. We find that the dissolution of nitrogen is much slower than the movement of the collapsing water front, and the re-expansion of the dense nitrogen droplet gives rise to bubble rebound. The incompatibility of the collapse and dissolution time scales should be considered for continuum-scale modeling of bubble dynamics. Finally, we also confirm that the diffusion coefficient for dissolved nitrogen is insensitive to pressure as the liquid transitions from a compressed to a stretched state.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2025

Uncertainty quantification and optimization of precipitating hydrometeor parameters for winter precipitation in a cloud microphysics scheme

The precipitating hydrometeor parameters used in cloud microphysics schemes carry inherent uncertainties. The quantification of these uncertainties, together with parameter optimization, can significantly improve precipitation forecasts. This study investigates the effects of 13 parameters in the Weather Research and Forecasting (WRF) Double-Moment 6-class (WDM6) microphysics scheme, which define the hydrometeor characteristics such as fall velocity–diameter and mass–diameter relationships, as well as the shape parameter of the drop size distribution for precipitating particles such as rain, snow, and graupel on simulated winter precipitation. A comparison between the model's pre-defined parameters and observations from the International Collaborative Experiments for the PyeongChang 2018 Olympic and Paralympic winter games (ICE-POP 2018) field campaign reveals that the fall velocity–diameter relationship for rain, the mass–diameter relationships for snow and graupel, and the shape parameters for all precipitating particles in the WDM6 scheme deviate from the median values observed by the two-dimensional video disdrometer (2DVD). To quantify parameter sensitivities, a perturbed parameter ensemble (PPE) of 256 simulations was conducted within parameter ranges constrained by 2DVD observations for three winter precipitation cases. Bayesian optimization was then applied to identify parameter sets that minimized the root mean square error (RMSE) for each case, achieving reductions of up to 30.2 %. These results demonstrate that ensemble-based uncertainty quantification and parameter optimization can help identify key parameters and provide a pathway to improving precipitation simulation performance. In addition, measurement sites can be strategically selected based on regions that show high sensitivity to variations in hydrometeor characteristic parameters.

Bayesian optimization↗
DOE OSTI2021

Assessing climate-change-induced flood risk in the Conasauga River watershed: an application of ensemble hydrodynamic inundation modeling

Abstract. This study evaluates the impact of potential future climate change on flood regimes, floodplain protection, and electricity infrastructures across the Conasauga River watershed in the southeastern United States through ensemble hydrodynamic inundation modeling. The ensemble streamflow scenarios were simulated by the Distributed Hydrology Soil Vegetation Model (DHSVM) driven by (1) 1981–2012 Daymet meteorological observations and (2) 11 sets of downscaled global climate models (GCMs) during the 1966–2005 historical and 2011–2050 future periods. Surface inundation was simulated using a GPU-accelerated Two-dimensional Runoff Inundation Toolkit for Operational Needs (TRITON) hydrodynamic model. A total of 9 out of the 11 GCMs exhibit an increase in the mean ensemble flood inundation areas. Moreover, at the 1 % annual exceedance probability level, the flood inundation frequency curves indicate a ∼ 16 km2 increase in floodplain area. The assessment also shows that even after flood-proofing, four of the substations could still be affected in the projected future period. The increase in floodplain area and substation vulnerability highlights the need to account for climate change in floodplain management. Overall, this study provides a proof-of-concept demonstration of how the computationally intensive hydrodynamic inundation modeling can be used to enhance flood frequency maps and vulnerability assessment under the changing climatic conditions.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Computationally efficient and error aware surrogate construction for numerical solutions of subsurface flow through porous media

Limiting the injection rate to restrict the pressure below a threshold at a critical location can be an important goal of simulations that model the subsurface pressure between injection and extraction wells. The pressure is approximated by the solution of Darcy’s partial differential equation for a given permeability field. The subsurface permeability is modeled as a random field since it is known only up to statistical properties. This induces uncertainty in the computed pressure. Solving the partial differential equation for an ensemble of random permeability simulations enables estimating a probability distribution for the pressure at the critical location. These simulations are computationally expensive, and practitioners often need rapid online guidance for real-time pressure management. An ensemble of numerical partial differential equation solutions is used to construct a Gaussian process regression model that can quickly predict the pressure at the critical location as a function of the extraction rate and permeability realization. The Gaussian process surrogate analyzes the ensemble of numerical pressure solutions at the critical location as noisy observations of the true pressure solution, enabling robust inference using the conditional Gaussian process distribution. Our first novel contribution is to identify a sampling methodology for the random environment and matching kernel technology for which fitting the Gaussian process regression model scales as O ( n log n ) instead of the typical O ( n 3 ) rate in the number of samples n used to fit the surrogate. The surrogate model allows almost instantaneous predictions for the pressure at the critical location as a function of the extraction rate and permeability realization. Our second contribution is a novel algorithm to calibrate the uncertainty in the surrogate model to the discrepancy between the true pressure solution of Darcy’s equation and the numerical solution. Finally, although our method is derived for building a surrogate for the solution of Darcy’s equation with a random permeability field, the framework broadly applies to solutions of other partial differential equations with random coefficients.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Seasonal and CO 2 -Induced Shifts of the ITCZ: Testing Energetic Controls in Idealized Simulations with Comprehensive Models

The Tropical Rain Belts with an Annual Cycle and Continent Model Intercomparison Project (TRACMIP) ensemble—a multimodel ensemble of slab-ocean simulations in idealized configurations—provides a test of the relationship between the zonal mean ITCZ and the cross-equatorial atmospheric energy transports (AHT eq ). In a gross sense, the ITCZ position is linearly related to AHT eq , as expected from the energetic framework. Yet, in many aspects, the TRACMIP model simulations do not conform to the framework. Throughout the annual cycle there are large excursions in the ITCZ position unrelated to changes in the AHT eq and, conversely, substantial variations in the magnitude of the AHT eq while the ITCZ is stationary at its northernmost position. Variations both in the net vertical energy input at the ITCZ location and in the vertical profile of ascent play a role in setting the model behavior apart from the conceptual framework. Nevertheless, a linear fit to the ITCZ–AHT eq relationship captures a substantial fraction of the seasonal variations in these quantities as well as the intermodel or across-climate variations in their annual mean values. The slope of the ITCZ–AHT eq linear fit for annual mean changes across simulations with different forcings and configurations varies in magnitude and even sign from model to model and we identify variations in the vertical profile of ascent as a key factor. A simple sea surface temperature–based index avoids the complication of changes in the vertical structure of the atmospheric circulation and provides a more reliable diagnostic for the ITCZ position.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2020

Integrated hydrogeophysical modelling and data assimilation for geoelectrical leak detection

Time-lapse electrical resistivity tomography (ERT) measurements provide indirect observations of hydrological processes in the Earth's shallow subsurface at high spatial and temporal resolution. ERT has been used in the past decades to detect leaks and monitor the evolution of associated contaminant plumes. Specifically, inverted resistivity images allow visualization of the dynamic changes in the structure of the plume. However, existing methods do not allow the direct estimation of leak parameters (e.g. leak rate, location, etc.) and their uncertainties. We propose an ensemble-based data assimilation framework that evaluates proposed hydrological models against observed time-lapse ERT measurements without directly inverting for the resistivities. Each proposed hydrological model is run through the parallel coupled hydro-geophysical simulation code PFLOTRAN-E4D to obtain simulated ERT measurements. The ensemble of model proposals is then updated using an iterative ensemble smoother. In this paper, we demonstrate the proposed framework on synthetic and field ERT data from controlled tracer injection experiments. Our results show that the approach allows joint identification of contaminant source location, initial release time, and solute loading from the cross-borehole time-lapse ERT data, alongside with an assessment of uncertainties in these estimates. We demonstrate a reduction in site-wide uncertainty by comparing the prior and posterior plume mass discharges at a selected image plane. This framework is particularly attractive to sites that have previously undergone extensive geological investigation (e.g., nuclear sites). It is well suited to complement ERT imaging and we discuss practical issues in its application to field problems.

58 GEOSCIENCES↗
DOE OSTI2025

Uncertainty of 21st Century western U.S. snowfall loss derived from regional climate model large ensemble

Abstract The western United States is dependent on winter snowfall over its major mountain ranges, which gradually melts each year, serving as a natural reservoir for water resources. In a future warmer climate, much of this snowfall could be replaced by rain, making it more challenging to capture and store water. In this study, we utilize an ensemble of dynamically downscaled simulations forced by 14 global climate models (GCMs). These GCMs project wildly different futures, in terms of both temperature and precipitation change, producing significant uncertainty in snowfall projections. Here we exploit the robust statistics of the downscaled ensemble, and diagose the sensitivity of end-of-century snowfall loss across the region to both warming and regional wetting/drying in the driving GCM. The windward slopes of the Sierra Nevada and Cascades are particularly sensitive to warming (losing ~ 15% annual snowfall per degree warming), with little influence of precipitation. By contrast, snowfall loss in the inter-mountain west is less sensitive to warming (~ 5% K −1 ), but is significantly offset/exacerbated by precipitation changes (~ 0.5% snow per 1% precipitation). Combining such sensitivities with the warming and regional precipitation signals in the full CMIP6 ensemble, we can fully quantify likely snowfall loss and its uncertainty at any location, for any emissions scenario. We find that the western U.S. as a whole will lose 34 ± 8% of its total volumetric snowfall by end-of-century under the high-emissions SSP3-7.0 scenario, but 25 ± 6% and 17 ± 6% under the lower-emissions SSP2-4.5 and SSP1-2.6 scenarios.

Norris, Jesse (ORCID:0000000289883326)↗
DOE OSTI2022

Accurate predictions from small boxes: variance suppression via the Zel'dovich approximation

Simulations have become an indispensable tool for accurate modelling of observables measured in galaxy surveys, but can be expensive if very large dynamic range in scale is required. We describe how to combine Lagrangian perturbation theory models with N-body simulations to reduce the effects of finite computational volume in the prediction of ensemble average properties in the simulations within the context of control variates. In particular we use the fact that Zel'dovich displacements, computed during initial condition generation for any simulation, correlate strongly with the final density field. Since all the correlators of biased tracers can be computed with arbitrary precision for these displacements, pairing the Zel'dovich `simulation' with the N-body realization allows hundredfold reductions in sample variance for power spectrum or correlation function estimation. Additionally, Zel'dovich control variates can accurately extend matter or tracer field emulators to larger scales than previously possible, as well as improving measurements of statistics in simulations which are inherently limited to small volumes, such as hydrodynamical simulations of galaxy formation and reionization.

79 ASTRONOMY AND ASTROPHYSICS↗
DOE OSTI2024

Portable, heterogeneous ensemble workflows at scale using libEnsemble

libEnsemble is a Python-based toolkit for running dynamic ensembles, developed as part of the DOE Exascale Computing Project. The toolkit utilizes a unique generator–simulator–allocator paradigm, where generators produce input for simulators, simulators evaluate those inputs, and allocators decide whether and when a simulator or generator should be called. The generator steers the ensemble based on simulation results. Generators may, for example, apply methods for numerical optimization, machine learning, or statistical calibration. libEnsemble communicates between a manager and workers. Flexibility is provided through multiple manager–worker communication substrates each of which has different benefits. These include Python’s multiprocessing, mpi4py, and TCP. Multisite ensembles are supported using Balsam or Globus Compute. We overview the unique characteristics of libEnsemble as well as current and potential interoperability with other packages in the workflow ecosystem. We highlight libEnsemble’s dynamic resource features: libEnsemble can detect system resources, such as available nodes, cores, and GPUs, and assign these in a portable way. These features allow users to specify the number of processors and GPUs required for each simulation; and resources will be automatically assigned on a wide range of systems, including Frontier, Aurora, and Perlmutter. Such ensembles can include multiple simulation types, some using GPUs and others using only CPUs, sharing nodes for maximum efficiency. We also describe the benefits of libEnsemble’s generator–simulator coupling, which easily exposes to the user the ability to cancel, and portably kill, running simulations based on models that are updated with intermediate simulation output. We demonstrate libEnsemble’s capabilities, scalability, and scientific impact via a Gaussian process surrogate training problem for the longitudinal density profile at the exit of a plasma accelerator stage. In conclusion, the study uses gpCAM for the surrogate model and employs either Wake-T or WarpX simulations, highlighting efficient use of resources that can easily extend to exascale.

Dynamic ensembles↗