Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “ensemble prediction”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 379 records · Page 21
All sourcesNASA NTRSDOE OSTI
DOE OSTI2025

Coupled Lake‐Atmosphere‐Land Physics Uncertainties in a Great Lakes Regional Climate Model

Abstract This study develops a surrogate‐based method to assess the uncertainty within a convective permitting integrated modeling system of the Great Lakes region, arising from interacting physics parameterizations across the lake, atmosphere, and land surface. Perturbed physics ensembles of the model during the 2018 summer are used to train a neural network surrogate model to predict lake surface temperature (LST) and near‐surface air temperature (T2m). Average physics uncertainties are determined to be 1.5C for LST and T2m over land, and 1.9C for T2m over lake, but these have significant spatiotemporal variations. We find that atmospheric physics parameterizations alone are the dominant sources of uncertainty (45%–53%), while lake and land parameterizations account for 33% and 38% of the uncertainty of LST and T2m over land respectively. Interactions of atmosphere physics parameterizations with those of the land and lake contribute to an additional 13%–17% of the total variance. LST and T2m over the lake are more uncertain in the deeper northern lakes, particularly during the rapid warming phase that occurs in late spring/early summer. The LST uncertainty increases with sensitivity to the lake model's surface wind stress scheme. T2m over land is more uncertain over forested areas in the north, where it is most sensitive to the land surface model, than the more agricultural land in the south, where it is most sensitive to the atmospheric planetary boundary and surface layer scheme. Uncertainty also increases in the southwest during multiday temperature declines with higher sensitivity to the land surface model.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

Wide range of possible trajectories of North Atlantic climate in a warming world

Decadal variability in the North Atlantic Ocean impacts regional and global climate, yet changes in internal decadal variability under anthropogenic radiative forcing remain largely unexplored. Here we use the Community Earth System Model 2 Large Ensemble under historical and the Shared Socioeconomic Pathway 3-7.0 future radiative forcing scenarios and show that the ensemble spread in northern North Atlantic sea surface temperature (SST) more than doubles during the mid-twenty-first century, highlighting an exceptionally wide range of possible climate states. Furthermore, there are strikingly distinct trajectories in these SSTs, arising from differences in the North Atlantic deep convection among ensemble members starting by 2030. We propose that these are stochastically triggered and subsequently amplified by positive feedbacks involving coupled ocean-atmosphere-sea ice interactions. Freshwater forcing associated with global warming seems necessary for activating these feedbacks, accentuating the impact of external forcing on internal variability. Further investigation on seven additional large ensembles affirms the robustness of our findings. By monitoring these mechanisms in real time and extending dynamical model predictions after positive feedbacks activate, we may achieve skillful long-lead North Atlantic decadal predictions that are effective for multiple decades.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2022

Interfacial water asymmetry at ideal electrochemical interfaces

Controlling electrochemical reactivity requires a detailed understanding of the charging behavior and thermodynamics of the electrochemical interface. Experiments can independently probe the overall charge response of the electrochemical double layer by capacitance measurements and the thermodynamics of the inner layer with potential of maximum entropy measurements. Relating these properties by computational modeling of the electrochemical interface has so far been challenging due to the low accuracy of classical molecular dynamics (MD) for capacitance and the limited time and length scales of ab initio MD. Here, we combine large ensembles of long-time-scale classical MD simulations with charge response from electronic density functional theory to predict the potential-dependent capacitance of a family of ideal aqueous electrochemical interfaces with different peak capacitances. We show that while the potential of maximum capacitance varies, this entire family exhibits an electrode charge of maximum capacitance (CMC) between -2.9 and -2.2 μC/cm 2 , regardless of the details in the electronic response. Simulated heating of the same interfaces reveals that the entropy peaks at a charge of maximum entropy (CME) of -5.1 ± 0.6 μC/cm 2 , in agreement with experimental findings for metallic electrodes. The CME and CMC both indicate asymmetric response of interfacial water that is stronger for negatively charged electrodes, while the difference between CME and CMC illustrates the richness in behavior of even the ideal electrochemical interface.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

Addressing uncertainty in genome-scale metabolic model reconstruction and analysis

The reconstruction and analysis of genome-scale metabolic models constitutes a powerful systems biology approach, with applications ranging from basic understanding of genotype-phenotype mapping to solving biomedical and environmental problems. However, the biological insight obtained from these models is limited by multiple heterogeneous sources of uncertainty, which are often difficult to quantify. Here we review the major sources of uncertainty and survey existing approaches developed for representing and addressing them. A unified formal characterization of these uncertainties through probabilistic approaches and ensemble modeling will facilitate convergence towards consistent reconstruction pipelines, improved data integration algorithms, and more accurate assessment of predictive capacity.

59 BASIC BIOLOGICAL SCIENCES↗
DOE OSTI2025

Exploring Data Set Bias and Decision Support with Predictive Uncertainty Through Bayesian Approximations and Convolutional Neural Networks

Individual seismic catalogs can contain multiscale observations from fault level to global scales and associated waveforms from discrete events reflect crustal structure across many different scales and locations. Seismic network aperture, geographic location, and observation distance may not provide informative guidance or intuition on how different catalogs will behave across models trained under different conditions. We rely on uncertainty to provide guardrails for when to trust model decisions, but understanding when our uncertainty is trustworthy is an open challenge. Here, in this work, we explore Bayesian approximation methods for assigning predictive uncertainty in seismic event classification problems. We find that computationally expensive Bayesian approximations do not outperform simple ensemble methods. We also find that when exploiting multiple seismic event catalogs, joint training with data from all the catalogs combined with Bayesian approximations and supervised training for classification can obscure bias and result in less robust uncertainty while also not providing substantial performance benefits compared to training individual models for each catalog.

58 GEOSCIENCES↗
DOE OSTI2020

Conservative Estimation of Tail Probabilities from Limited Sample Data

Several sparse-sample uncertainty quantification (UQ) methods are compared for conservative but not overly conservative estimation of small tail probabilities involving responses that lay beyond specified thresholds in the tails of probability distributions. Sixteen very differently shaped distributions (or probability density functions, PDFs) and tail probability magnitudes ranging from 10 -5 to 10 -1 are considered in order for the study to be relevant to a wide range of risk analysis and quantification of margins and uncertainty (QMU) problems. The emphasis of the study is on limited data regimes ranging from N = 2 to 20 samples, reflective of most experimental and some expensive computational situations. Relatively simple sparse-sample UQ methods tested for this regime involve statistical tolerance interval "Equivalent Normal and related "Ensemble of Normals" and "Superdistribution (SD) approaches. (The independently derived SD is effectively equivalent to the Bayesian posterior predictive distribution given the assumptions of the derivation.) The performance of the methods was generally improved for N ≥ 5 samples with a generalized Jackknife resampling technique, which determines a tail probability estimate by averaging estimates from smaller sub-samples. Several quantitative metrics for method conservatism and accuracy of tail probability estimation are used to assess and rank the methods' performance over many random trials for each test PDF and probability magnitude. A variant of Bootstrap resampling was also tried, but did not significantly improve tail probability estimates in most cases. Detailed results are presented from over 100-million tests over the above factors that provide useful granular information on which methods or combination of methods perform best in various areas of the factor space.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Examples of Mission-driven Data Science from Jefferson Lab and ACES

This presentation details mission-driven data science initiatives at Jefferson Lab and the Joint Institute for Advanced Computing on Environmental Studies (ACES). JLab, a U.S. Department of Energy Office of Science national laboratory, operates the Continuous Electron Beam Accelerator Facility (CEBAF), and is the lead institute for the new High Performance Data Facility (HPDF) Hub. The Joint Institute for ACES brings together interdisciplinary teams in health informatics, climate modeling, computer science, and physics to address environmental challenges, including flood modeling. The Hampton Roads region, particularly Norfolk and Virginia Beach, faces increasing flood risks, motivating the need for rapid, reliable, and risk-aware decision support. ACES’s flooding work has a focus on uncertainty quantification (UQ) and machine learning (ML) for coastal flood management. The work is motivated by the increasing vulnerability of communities such as Norfolk and Virginia Beach, Virginia, to frequent coastal flooding events, and the need for rapid, reliable decision support. The research develops computationally efficient ML surrogate models to forecast water levels and flooding risk. A central theme is the quantification and calibration of predictive uncertainty, especially for out-of-distribution (OOD) scenarios, using techniques such as Monte Carlo Dropout, Deep Ensembles, Gaussian Processes, and Deep Quantile Regression (DQR). The study demonstrates that distance-aware UQ is critical for reliable scientific AI, particularly in high-dimensional, safety-critical, and real-time applications.

McSpadden, Diana [Thomas Jefferson National Accele↗
DOE OSTI2026

Machine-learning-based estimates of global natural vegetated wetland methane emissions (2000–2025)

Wetlands are the largest natural source of atmospheric methane (CH 4 ), yet comprehensive global budgets are typically delayed by years, preventing a timely understanding of CH 4 sources, sinks, and trends. To reduce this delay, we present a model emulator-driven framework and accompanying workflow that enable timely, continuous emission updates using a machine-learning emulator to reconstruct spatially explicit monthly emission fields at 1° × 1° resolution. We apply this framework to a global dataset of natural vegetated wetland CH 4 emissions to extend the most recent Global Methane Budget (GMB; Saunois et al., 2025) record that covers the 2000–2020 emissions through 2025. In the test data (∼ 30 % of the total dataset), the emulator achieved a global R 2 of 0.65 ± 0.003 (mean ± 95 % CI, hereafter) and an RMSE of 5.49 ± 0.12×10 -3 Tg CH 4 yr −1 . The emulator is trained on 35 GMB model estimates, including 22 process-based models and 13 atmospheric inversions, paired with 10 ensemble realizations of 11 gridded climate predictor variables from atmospheric reanalyses. Our results show that the global mean predicted wetland CH 4 emissions for 2021–2025 (157.8 ± 2.4 Tg CH 4 yr −1 ) are not significantly higher (∼ 0.05 Tg CH 4 yr −1 ) than the 2000–2020 baseline. However, this stability masks a significant hemispheric redistribution of emissions. We detect an increase in Northern Hemisphere (NH) emissions in 2021–2025, with mid- and high-latitudes increasing by 0.76 ± 0.07 and 0.35 ± 0.03 Tg CH 4 yr −1 , respectively, while the tropics and Southern Hemisphere (SH) extratropics show offsetting negative trends (−0.95 ± 0.19 and -0.11 ± 0.02 Tg CH 4 yr −1 , respectively). The predicted emissions are able to capture the low emissions in 2023 in South America linked to El Niño-related drought, as reported by recent studies (Ciais et al., 2026; Quinn et al., 2025). Furthermore, we identify a distinct seasonal amplification of global emission trends that peaks in late boreal summer. This new modeled dataset and operational framework bridge the gap between the latest updated budgets and low-latency monitoring, providing a scalable capacity to frequently update global emission estimates and critical early warnings of regional wetland feedback loops. The data are publicly available at https://doi.org/10.5281/zenodo.18870108 (Li et al., 2026).

Li, Mengze [National University of Singapore (Sing↗
DOE OSTI2021

Statistical emulation of a perturbed basal melt ensemble of an ice sheet model to better quantify Antarctic sea level rise uncertainties

Abstract. Antarctic ice shelves are vulnerable to warming ocean temperatures, and some have already begun thinning in response to increased basal melt rates. Sea level is therefore expected to rise due to Antarctic contributions, but uncertainties in its amount and timing remain largely unquantified. In particular, there is substantial uncertainty in future basal melt rates arising from multi-model differences in thermal forcing and how melt rates depend on that thermal forcing. To facilitate uncertainty quantification in sea level rise projections, we build, validate, and demonstrate projections from a computationally efficient statistical emulator of a high-resolution (4 km) Antarctic ice sheet model, the Community Ice Sheet Model version 2.1. The emulator is trained to a large (500-member) ensemble of 200-year-long 4 km resolution transient ice sheet simulations, whereby regional basal melt rates are perturbed by idealized (yet physically informed) trajectories. The main advantage of our emulation approach is that by sampling a wide range of possible basal melt trajectories, the emulator can be used to (1) produce probabilistic sea level rise projections over much larger Monte Carlo ensembles than are possible by direct numerical simulation alone, thereby providing better statistical characterization of uncertainties, and (2) predict the simulated ice sheet response under differing assumptions about basal melt characteristics as new oceanographic studies are published, without having to run additional numerical ice sheet simulations. As a proof of concept, we propagate uncertainties about future basal melt rate trajectories, derived from regional ocean models, to generate probabilistic sea level rise estimates for 100 and 200 years into the future.

97 MATHEMATICS AND COMPUTING↗
DOE OSTI2025

Physics-based hybrid machine learning for critical heat flux prediction with uncertainty quantification

Critical heat flux (CHF) is a key quantity in nuclear system modeling due to its impact on heat transfer, safety margins, and reactor performance. This study develops and validates an uncertainty-aware hybrid modeling approach that combines machine learning with physics-based models to predict CHF in cases of dryout. The Biasi and Bowring empirical correlations were paired with three ML uncertainty quantification (UQ) techniques: deep neural network (DNN) ensembles, Bayesian neural networks (BNNs), and deep Gaussian processes (DGPs). A pure ML model without a base model was evaluated for comparison. Model performance was assessed under plentiful (7,350 points) and limited (9 points) training data scenarios using parity, uncertainty distributions, and calibration curves. Results show that the Biasi hybrid DNN ensemble achieved the best overall performance, with a mean absolute relative error of 1.846%, and well-calibrated uncertainty estimates. The BNN-based hybrids showed slightly higher error (2.14%) but superior uncertainty calibration. DGP models underperformed, with over 6% error and poor uncertainty calibration. All hybrid models outperformed pure machine learning configurations, demonstrating resistance against data scarcity. These findings indicate that hybrid modeling significantly improves predictive accuracy, interpretability, and resilience to data scarcity. The integration of uncertainty awareness provides actionable confidence in CHF predictions, which is vital for safety-critical decisions in nuclear applications. This hybrid approach offers a viable pathway for deploying ML models in reactor analysis tools while preserving domain knowledge and physical consistency.

22 GENERAL STUDIES OF NUCLEAR REACTORS↗
DOE OSTI2024

Bridging Hydrological Ensemble Simulation and Learning Using Deep Neural Operators

Ensemble-based simulation and learning (ESnL) has long been used in hydrology for parameter inference, but computational demands of process-based ESnL can be quite high. To address this issue, we propose a deep neural operator learning approach. Neural operators are generic machine learning algorithms that can learn functional mappings between infinite-dimensional spaces, providing a highly flexible tool for scientific machine learning. Our approach is built upon DeepONet, a specific deep neural operator, and is designed to address several common problems in hydrology, namely, model parameter estimation, prediction at ungaged locations, and uncertainty quantification. Here we demonstrate the effectiveness of our DeepONet-based workflow using an existing large model ensemble created for an eastern U.S. watershed that is instrumented with 10 streamflow gages. Results suggest DeepONet achieves high efficiency in learning an ML surrogate model from the model ensemble, with the modified Kling-Gupta Efficiency exceeding 0.9 on holdout test sets. Parameter inference, carried out using the trained DeepONet surrogate model and genetic algorithm, also yields robust results. Additionally, we formulate and train a separate DeepONet model for physics-informed, seq-to-seq streamflow forecasting, which further reduces biases in the pre-trained DeepONet surrogate model. While this study focuses primarily on a single watershed, our approach is general and may be extended to enable learning from model ensembles across multiple basins or models. Thus, this research represents a significant contribution to the application of hybrid machine learning in hydrology.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2025

Computationally efficient subglacial drainage modelling using Gaussian process emulators: GlaDS-GP v1.0

Subglacial drainage models represent water flow at the ice–bed interface through coupled distributed and channelized systems to determine water pressure, discharge, and drainage system geometry. While they are used to understand processes such as the relationship between surface melt and ice flow, the number of uncertain model parameters and the computational cost of running models makes it difficult to adequately explore the high-dimensional parameter space and evaluate uncertainty in model predictions. Here, we develop Gaussian process (GP) emulators that make fast predictions with associated uncertainty of subglacial drainage model outputs. Using a truncated principal component (PC) basis representation, we construct a GP emulator for diurnally averaged subglacial water pressure. We also explore emulation of scalar variables describing drainage efficiency and configuration. We train the emulators using ensembles of up to 512 simulations varying eight parameters of the Glacier Drainage System (GlaDS) model on a synthetic domain intended to represent an ice-sheet margin. The emulators make predictions ∼ 1000 times faster than GlaDS simulations, with errors <3 % for the water pressure field and ∼ 5 %–9 % for drainage efficiency and configuration. We apply the emulators to explore the eight-dimensional parameter space by computing variance-based parameter sensitivity indices, finding that three parameters (ice flow coefficient, bed bump aspect ratio, and the subglacial cavity system conductivity) explain 90 % of the variance in modelled water pressure in response to parameter changes. The GP emulator approach described here is well suited to integrating observational data with models to make calibrated, credible predictions of subglacial drainage.

58 GEOSCIENCES↗
DOE OSTI2023

Evaluating skill in predicting the Interdecadal Pacific Oscillation in initialized decadal climate prediction hindcasts in E3SMv1 and CESM1 using two different initialization methods and a small set of start years

Abstract It is a daunting challenge to conduct initialized hindcasts with enough ensemble members and associated start years to form a drifted climatology from which to compute the anomalies necessary to quantify the skill of the hindcasts when compared to observations. This limits the ability to experiment with case studies and other applications where only a few initial years are needed. Here we run a set of hindcasts with CESM1 and E3SMv1 using two different initialization methods for a limited set of start years and use the respective uninitialized free-running historical simulations to form the model climatologies. Since the drifts from the observed initial states in the hindcasts toward the uninitialized model state are large and rapid, after a few years the drifted initialized models approach the uninitialized model climatological errors. Therefore, hindcasts from the limited start years can use the uninitialized climatology to represent the drifted model states after about lead year 3, providing a means to compute forecast anomalies in the absence of a large hindcast sample. There is comparable skill for predicting spatial patterns of multi-year Pacific sea surface temperature anomalies in the domain of the Interdecadal Pacific Oscillation using this method compared to the conventional methodology with a large hindcast data set, though there is a model dependence to the drifts in the two initialization methods.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2024

A Bayesian Multi-fidelity Neural Network to Predict Nonlinear Frequency Backbone Curves

The use of structural mechanics models during the design process often leads to the development of models of varying fidelity. Often low-fidelity models are efficient to simulate but lack accuracy, while the high-fidelity counterparts are accurate with less efficiency. Here, this paper presents a multi-fidelity surrogate modeling approach that combines the accuracy of a high-fidelity finite element model with the efficiency of a low-fidelity model to train an even faster surrogate model that parameterizes the design space of interest. The objective of these models is to predict the nonlinear frequency backbone curves of the Tribomechadynamics Research Challenge benchmark structure which exhibits simultaneous nonlinearities from frictional contact and geometric nonlinearity. The surrogate model consists of an ensemble of neural networks that learn the mapping between low and high-fidelity data through nonlinear transformations. Bayesian neural networks are used to assess the surrogate model's uncertainty. Once trained, the multi-fidelity neural network is used to perform sensitivity analysis to assess the influence of the design parameters on the predicted backbone curves. Additionally, Bayesian calibration is performed to update the input parameter distributions to correlate the model parameters to the collection of experimentally measured backbone curves.

42 ENGINEERING↗
DOE OSTI2024

Multilevel Monte Carlo methods for the Grad-Shafranov free boundary problem

The equilibrium configuration of a plasma in an axially symmetric reactor is described mathematically by a free boundary problem associated with the celebrated Grad-Shafranov equation. The presence of uncertainty in the model parameters introduces the need to quantify the variability in the predictions. This is often done by computing a large number of model solutions on a computational grid for an ensemble of parameter values and then obtaining estimates for the statistical properties of solutions. In this study, we explore the savings that can be obtained using multilevel Monte Carlo methods, which reduce costs by performing the bulk of the computations on a sequence of spatial grids that are coarser than the one that would typically be used for a simple Monte Carlo simulation. We examine this approach using both a set of uniformly refined grids and a set of adaptively refined grids guided by a discrete error estimator. Numerical experiments show that multilevel methods dramatically reduce the cost of simulation, with cost reductions typically on the order of 60 or more and possibly as large as 200. Furthermore, adaptive griding results in more accurate computation of geometric quantities such as x-points associated with the model.

71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSIC↗
DOE OSTI2022

Ensemble-function relationships to dissect mechanisms of enzyme catalysis

Decades of structure-function studies have established our current extensive understanding of enzymes. However, traditional structural models are snapshots of broader conformational ensembles of interchanging states. We demonstrate the need for conformational ensembles to understand function, using the enzyme ketosteroid isomerase (KSI) as an example. Comparison of prior KSI cryogenic x-ray structures suggested deleterious mutational effects from a misaligned oxyanion hole catalytic residue. However, ensemble information from room-temperature x-ray crystallography, combined with functional studies, excluded this model. Ensemble-function analyses can deconvolute effects from altering the probability of occupying a state (P-effects) and changing the reactivity of each state (k-effects); our ensemble-function analyses revealed functional effects arising from weakened oxyanion hole hydrogen bonding and substrate repositioning within the active site. Ensemble-function studies will have an integral role in understanding enzymes and in meeting the future goals of a predictive understanding of enzyme catalysis and engineering new enzymes.

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CH↗
DOE OSTI2021

A clustering-based approach to ocean model–data comparison around Antarctica

The Antarctic Continental Shelf seas (ACSS) are a critical, rapidly changing element of the Earth system. Analyses of global-scale general circulation model (GCM) simulations, including those available through the Coupled Model Intercomparison Project, Phase 6 (CMIP6), can help reveal the origins of observed changes and predict the future evolution of the ACSS. However, an evaluation of ACSS hydrography in GCMs is vital: previous CMIP ensembles exhibit substantial mean-state biases (reflecting, for example, misplaced water masses) with a wide inter-model spread. Because the ACSS are also a sparely sampled region, grid-point-based model assessments are of limited value. Our goal is to demonstrate the utility of clustering tools for identifying hydrographic regimes that are common to different source fields (model or data), while allowing for biases in other metrics (e.g., water mass core properties) and shifts in region boundaries. We apply K-means clustering to hydrographic metrics based on the stratification from one GCM (Community Earth System Model version 2; CESM2) and one observation-based product (World Ocean Atlas 2018; WOA), focusing on the Amundsen, Bellingshausen and Ross seas. When applied to WOA temperature and salinity profiles, clustering identifies “primary” and “mixed” regimes that have physically interpretable bases. For example, meltwater-freshened coastal currents in the Amundsen Sea and a region of high-salinity shelf water formation in the southwestern Ross Sea emerge naturally from the algorithm. Both regions also exhibit clearly differentiated inner- and outer-shelf regimes. The same analysis applied to CESM2 demonstrates that, although mean-state model biases in water mass T–S characteristics can be substantial, using a clustering approach highlights that the relative differences between regimes and the locations where each regime dominates are well represented in the model. CESM2 is generally fresher and warmer than WOA and has a limited fresh-water-enriched coastal regimes. Given the sparsity of observations of the ACSS, this technique is a promising tool for the evaluation of a larger model ensemble (e.g., CMIP6) on a circum-Antarctic basis.

54 ENVIRONMENTAL SCIENCES↗
DOE OSTI2021

Explaining and predicting human behavior and social dynamics in simulated virtual worlds: reproducibility, generalizability, and robustness of causal discovery methods

Ground Truth program was designed to evaluate social science modeling approaches using simulation test beds with ground truth intentionally and systematically embedded to understand and model complex Human Domain systems and their dynamics Lazer et al. (Science 369:1060–1062, 2020). Our multidisciplinary team of data scientists, statisticians, experts in Artificial Intelligence (AI) and visual analytics had a unique role on the program to investigate accuracy, reproducibility, generalizability, and robustness of the state-of-the-art (SOTA) causal structure learning approaches applied to fully observed and sampled simulated data across virtual worlds. In addition, we analyzed the feasibility of using machine learning models to predict future social behavior with and without causal knowledge explicitly embedded. In this paper, we first present our causal modeling approach to discover the causal structure of four virtual worlds produced by the simulation teams—Urban Life, Financial Governance, Disaster and Geopolitical Conflict. Our approach adapts the state-of-the-art causal discovery (including ensemble models), machine learning, data analytics, and visualization techniques to allow a human-machine team to reverse-engineer the true causal relations from sampled and fully observed data. We next present our reproducibility analysis of two research methods team’s performance using a range of causal discovery models applied to both sampled and fully observed data, and analyze their effectiveness and limitations. We further investigate the generalizability and robustness to sampling of the SOTA causal discovery approaches on additional simulated datasets with known ground truth. Our results reveal the limitations of existing causal modeling approaches when applied to large-scale, noisy, high-dimensional data with unobserved variables and unknown relationships between them. We show that the SOTA causal models explored in our experiments are not designed to take advantage from vasts amounts of data and have difficulty recovering ground truth when latent confounders are present; they do not generalize well across simulation scenarios and are not robust to sampling; they are vulnerable to data and modeling assumptions, and therefore, the results are hard to reproduce. Finally, when we outline lessons learned and provide recommendations to improve models for causal discovery and prediction of human social behavior from observational data, we highlight the importance of learning data to knowledge representations or transformations to improve causal discovery and describe the benefit of causal feature selection for predictive and prescriptive modeling.

97 MATHEMATICS AND COMPUTING↗