Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Intelligent Systems”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 1,153 records · Page 64

Artificial Intelligence for Natural Gas Utilities: A Primer

Modern natural gas utilities face numerous challenges and competing priorities from various stakeholders. Policymakers, customers, and advocacy groups want to see gas utilities improve performance on safety, reliability, resilience, affordability, and environmental stewardship. State utility regulators — public utility commissions — are responsible for overseeing utility performance, ensuring that ratepayer funds are being spent in the public interest, and aligning utility goals with public goals. The use of new technologies is critical to enabling cost-effective performance on these attributes. Artificial intelligence (AI) is a widely used term among utilities and regulators, but the term means different things to different stakeholders, and it is often used to describe data analytics approaches that fall short of the formal definition of AI, which is: “…the ability of a machine to receive inputs and produce a behavior or reaction similar to that of an intelligent human being.” AI (and related tools, techniques, and technologies) can help utilities solve current and emerging challenges. By combining customer and system data with analytical tools and technologies, AI can augment human decision-makers by assisting in identifying problems and events before they occur, enabling resources to be more efficiently directed across utility infrastructure. The intended primary audience for this primer is state regulators, although utilities and other stakeholders might also find it useful and relevant to improve their awareness of AI. The objectives of this primer are to: (a) offer a set of broadly applicable definitions for AI and related terms, allowing regulators, utilities, and other stakeholders to speak the same language; (b) discuss how AI is currently being implemented in the gas utility sector; and (c) understand the challenges affecting AI solutions and how tools might be implemented in the future. This primer fits within NARUC’s goals of providing impartial information to improve the ability of public utility commissions to regulate in the public interest. As such, this primer does not seek to recommend AI over any other investment, nor does it endorse any particular vendor, product, or approach. It does seek to prepare state regulators to oversee AI investments by sharing information about the current landscape of commercially available tools. To these ends, the primer is organized as follows: Section I discusses the current environment in which natural gas utilities operate and how AI, when thoughtfully designed and implemented, can enable utilities to achieve performance goals; Section II offers definitions of AI and related terms within the data analytics discipline; Section III provides three current opportunities for which AI can offer solutions: replacing aging gas distribution infrastructure, preventing excavator damage to gas distribution infrastructure, and improving energy efficiency programs. This section discusses each problem statement in detail. Second, Section III includes a discussion of how costs and benefits of investments to solve each problem are measured. And third, this section offers real-world examples of utility implementation of AI solutions; Section IV discusses challenges with implementing AI, both from the perspective of utilities and regulators; Section V suggests areas in which AI could feasibly be implemented in the near future; Finally, Section VI offers concluding thoughts and areas for further research.

03 NATURAL GAS↗

Cyber Attack Resilient HVDC System (CARDS) (Final Scientific/Technical Report)

This report summarizes the activities conducted under the DOE-OE funded project DE-OE0000824, where ABB Inc. (ABB), in collaboration with University of Illinois at Urbana-Champaign (UIUC), Bonneville Power Administration (BPA), Argonne National Laboratories (ANL), and University of Idaho (UI) pursued the development of a system of defense for High Voltage Direct Current transmission systems against cyber-attacks (CARDS). An HVDC substation with CARDS features will be more capable of mitigating cyber-attacks especially those that seek to control or interrupt the power flow in the HVDC line. It empowers existing HVDC intelligent electronic devices to collaborate in identifying and blocking malicious intents to trip HVDC AC circuit breakers, block thyristor valves, misconfigure devices settings, even though the commands and the measurements comply with correct syntax. The CARDS functions utilize the physics of electromagnetic AC and DC systems, electric power engineering principles, and state estimation to bring more in-depth cyber defense closer to the protected HVDC substation devices. With the aim of advancing the state of the art, the team proceeded to develop a demonstrable system against cyber-attacks in an HVDC substation environment, via functional upgrades to existing firmware of commercial off-the-shelf HVDC station devices, and/or embedded hardware, without the need for additional instrumentation nor new instrumentation technologies. It is to ensure that the solutions resulting from the project will support the HVDC devices currently installed in the US electricity delivery infrastructure.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Summary Report on Synthetic Time History Development and Deployment

This report summarizes research work performed at the Colorado School of Mines (CSM) and George Washington University (GWU) in support of the DOE Office of Nuclear Energy Integrated Energy Systems (IES) program. In particular, the work focuses on improving and strengthening two fundamental steps in the mathematical and computational framework that is currently being developed to optimize the economic performance of Integrated Energy Systems. One of the innovative approaches taken by the IES program is that, rather than optimizing the system for a specific year or week or day, the optimization is performed using stochastic synthetic data. Synthetic data is generated by training artificial intelligence (AI) using available but limited data sets. The synthetic data should possess the same patterns as the original set and the same statistical properties, but never repeat the same set of values exactly. The work from CSM focuses on this first step; namely, investigating a methodology to determine the quality of the synthetic data and the right settings for properly training the AI. Once a large set of trustworthy data can be generated the problem of stochastic optimization can be tackled. The optimal IES configuration is highly dependent on the rare high peaks or deep depths of the net demand (grid electricity demand less variable renewable energy [VRE] production). Using an unbiased random sampling of the synthetic data, these extremes are very difficult to generate and therefore the optimization can be unrepresentative and inaccurate. GWU has started an approach to increase the likelihood of generating these rare events with the proper associated probabilistic weight. Both of these projects represent a step forward in achieving the capability to properly size the IESs and to properly capture the value that these systems can provide to the grid by absorbing volatility of net demand without disproportionately increasing the cost of maintaining grid reliability.

29 ENERGY PLANNING, POLICY, AND ECONOMY↗

Enabling Capabilities and Resources: 2024 Principal Investigator Meeting Proceedings

As a major supporter of basic genome-enabled research, BER’s Biological Systems Science Division (BSSD) fosters scientific discovery by funding - fundamental biological research across disciplines in conjunction with enabling investigational tools and computational capabilities that include world-class user facilities. The overarching goal of BSSD is to provide the necessary fundamental science to understand, predict, manipulate, and design biological systems that underpin innovations for bioenergy and bioproduct production and enhance understanding of natural, DOE-relevant environmental processes (Biological Systems Science Division Strategic Plan, 2021). To accelerate the U.S. bioeconomy, BSSD pursues innovative science underpinning advances in sustainable biofuels and bioproducts and the development of next-generation technologies and computational resources for systems biology research. The 2024 BSSD Enabling Capabilities and Resources (ECR) Principal Investigator (PI) meeting brought together PIs across the BSSD ECR portfolio to confer on shared interests and opportunities. The meeting was held concurrently with the Genomic Science program (GSP) PI meeting to optimize collaboration on research to advance bioenergy and the bioeconomy. Rick Stevens of Argonne National Laboratory gave a keynote on How Generative Artificial Intelligence Can Impact Biological Research (see Keynote: How Generative Artificial Intelligence Can Impact Biological Research, this page). Plenary presentations included several joint sessions that illuminated the integration and understanding of the larger BSSD mission. GSP’s objective is to provide systems-level understanding of plants, microbes, and their communities through its Bioenergy Research, Biosystems Design, and Environmental Microbiome Research portfolios. The objective of the ECR portfolio is to support development of computational and instrumental platforms to advance fundamental GSP research—and BER more broadly— toward the overall goal of understanding the functional principles of living systems and their response to environmental challenges.

59 BASIC BIOLOGICAL SCIENCES↗

Potential Ideas for an El Capitan Center of Excellence (COE) Around Intelligent Simulation

Proposing a Center of Excellence under the CORAL-2 NRE is a mandatory requirement. The current Sierra COE (CORAL-1) is largely focused on helping LLNL applications make the disruptive transition to a heterogeneous GPU-based system by 2018. We expect a continuation of COE activities around optimizing our broad and diverse application base (of so-called “traditional” simulation codes) optimized for the El Capitan architecture, as well as supporting the underlying software stack (compilers, tools, programming models, etc.) – but do not expect this to require as much effort in the El Capitan COE (assuming a heterogeneous node architecture). What follows are some thoughts on 3 potential topics of interest for an El Capitan COE at LLNL – largely focused around AI and machine learning and the concept of advancing our goal of intelligent simulation or cognitive computing in the timeframe of deployment and production use of El Capitan (2023-2038). We believe vendor engagement through a COE in this area will provide a natural point-of-interest between LLNL and our vendor partner for common advancement of machine learning capabilities focused on scientific data – potentially greatly broadening the ecosystem around HPC architectures and the supporting software stack for scientific simulation-based AI.

97 MATHEMATICS AND COMPUTING↗

Optimization and Prediction of Spectral Response of Metasurfaces Using Artificial Intelligence

Hot-electron generation has been a topic of intense research for decades for numerous applications ranging from photodetection and photochemistry to biosensing. Recently, the technique of hot-electron generation using non-radiative decay of surface plasmons excited by metallic nanoantennas, or meta-atoms, in a metasurface has attracted attention. These metasurfaces can be designed with thicknesses on the order of the hot-electron diffusion length. The plasmonic resonances of these ultrathin metasurfaces can be tailored by changing the shape and size of the meta-atoms. One of the fundamental mechanisms leading to generation of hot-electrons in such systems is optical absorption, therefore, optimization of absorption is a key step in enhancing the performance of any metasurface based hot-electron device. Here we utilized an artificial intelligence-based approach, the genetic algorithm, to optimize absorption spectra of plasmonic metasurfaces. Using genetic algorithm optimization strategies, we designed a polarization insensitive plasmonic metasurface with 90% absorption at 1550 nm that does not require an optically thick ground plane. We fabricated and optically characterized the metasurface and our experimental results agree with simulations. Finally, we present a convolutional neural network that can predict the absorption spectra of metasurfaces never seen by the network, thereby eliminating the need for computationally expensive simulations. Our results suggest a new direction for optimizing hot-electron based photodetectors and sensors.

77 NANOSCIENCE AND NANOTECHNOLOGY↗

Enhanced Oblique Decision Tree Enabled Policy Extraction for Deep Reinforcement Learning in Power System Emergency Control

Deep reinforcement learning (DRL) algorithms have successfully solved many challenging problems in various power system control scenarios. However, their decision-making process is usually regarded as black-boxes. Furthermore, how DRL models interact with human intelligence remains an open problem. Thus, this paper proposes a policy extraction framework to extract a complex DRL model into an explainable policy. This framework includes three parts: 1) DRL training and data generation. We train an agent for a specific control task and generate data, which contains the control policy of the agent. 2) Policy extraction. We propose an information gain rate based weighted oblique decision tree (IGR-WODT) for DRL policy extraction. 3) Policy evaluation. We define three metrics to evaluate the performance of the proposed approach. A case study for the under-voltage load shedding problem shows that the IGR-WODT presents a performance enhancement compared with DRL, weighted oblique decision tree, and univariate decision tree. The proposed policy extraction method could provide an intuitive explanation of the neural network decision-making process to the dispatchers when making final decisions on power grid operation. Also, the resulted rule-based controller could replace the deep neural network-based controller in many field edge devices with limited computing resources, providing comparable performance.

deep reinforcement learning↗

Machine Learning Using Open Data Sources for Detection of Nuclear Proliferation Activities (U)

In FY2020, Savannah River National Laboratory (SRNL) in collaboration with the Sanghani Center for Artificial Intelligence and Data Analytics (SCAIDA) at Virginia Polytechnic Institute and State University (VT) and funded by the Department of Energy’s (DOE) Defense Nuclear Nonproliferation Research and Development, began developing a demonstration prototype system that uses multiple machine learning and data analytic methods on large-scale open data sources to identify new, developing, and/or undeclared nuclear programs. Using the announcement in May 2018 of the proposed Savannah River Plutonium Processing Facility (SRPPF) as a test subject, the goal of this 2-year project is to forecast the SRPPF using only data prior to May 2018. The project work is split into a preliminary prototype development for the first year with an initial evaluation of viability followed by the second year of development to create an integrated prototype system and more extensive performance evaluation. This report documents the results of the preliminary-phase tasks.

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL P↗

Machine Learning Approaches for High-resolution Urban Land Cover Classification: A Comparative Study

The proliferation of several machine learning approaches makes it difficult to identify a suitable classification technique for analyzing high-resolution remote sensing images. In this study, ten classification techniques were compared from five broad machine learning categories. Surprisingly, the performance of simple statistical classification schemes like maximum likelihood and Logistic regression over complex and recent techniques is very close. Given that these two classifiers require little input from the user, they should still be considered for most classification tasks. Multiple classifier systems is a good choice if the resources permit.

97 MATHEMATICS AND COMPUTING↗

High-efficiency frost and ice control via sensing-assisted nanovibrational slippery surfaces

Frost and ice accretions on surfaces pose persistent challenges across numerous industrial, residential and transportation systems. While various removal strategies exist, they often suffer from limited effectiveness or high energy consumption, such as frosting delay, ice crack generation, and Joule heating. Here, in this work, we report a novel integrated approach combining vibrational quasi-liquid surface (QLS) and capacitive sensing for efficient condensate, frost, and ice management. Compared to Joule heating, our approach does not rely on complete melting and evaporation for removal, resulting in 68% and 95% energy savings for frost and ice removal, respectively. Our QLS coating significantly reduces surface retention forces, achieving 91% and 87% less residual mass compared to hydrophilic surfaces for frost and ice removal through surface nanovibration, respectively. The integrated capacitive sensor provides real-time detection of different phase states, enabling on-demand removal in precise timeframes. This sensor-assisted approach showed 3.8 times lower energy consumption compared to conventional Joule heating for defrosting. This synergistic integration of surface engineering, nanovibration, and intelligent sensing represents a significant advancement in phase change processes, offering an energy-efficient solution for frost and ice mitigation in energy-intensive systems.

Shen, Yuchen [Univ. of Texas at Dallas, Richardson↗

An Integrated ML/AI Framework for Digitizing, Structuring and Searching DOE U-TRU-Fuels Data with Gap Analysis of Non-DOE Records

The U.S. Department of Energy (DOE) Advanced Fuels Campaign (AFC) is advancing transmutation fuel technologies to reduce long-lived radioactive waste by converting minor actinides into shorter-lived or stable elements through irradiation in sodium-cooled fast reactors. Key experiments such as AFC-1, AFC-2, FUels for the transmutation of Trans-URanium elements In phéniX (FUTURIX)-Fortes Teneurs en Actinides (FTA), and Experimental Breeder Reactor-II (EBR-II) X501 have provided fuel fabrication, irradiation, and performance data on various transuranic-bearing fuel forms. This report documents the creation of an artificial-intelligence assisted database, which has consolidated all DOE-owned data related to Transuranic (TRU)-bearing fuel experiments and stored across it across both the Idaho National Laboratory (INL) Nuclear Data Management and Analysis System and the INL high performance computing (HPC) infrastructure. A dedicated webpage, hosted on the INL HPC system, has been developed to support role-based access and data interaction. The database architecture allows researchers to navigate large, heterogeneous archives with far greater speed and accuracy than manual search and lays the foundation for future expansion into multimodal nuclear materials analysis environments. The database represents a major step towards a nationally integrated fuels database utilizing artificial intelligence tools.

11 - NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Sensors with Intelligent Measurement Platform for Monitoring and Control in Distribution Circuits

This paper presents a medium-voltage intelligent measurement platform that provides post-processing of raw measurements received from commercially-available voltage/current sensors to enhance accuracy and bandwidth of the monitoring system. Furthermore, this platform integrates into a Data Distribution Service (DDS) messaging bus for distributed control and interoperability. The authors present the design, specifications, and major components included in the platform, and argue for applications and use cases that most benefit from this new approach. A laboratory testbed has been created to demonstrate the functionality, for which test results are presented for selected test cases.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Intelligent Sun Tracking for a CPV Power Plant

The output of a solar panel is strongly dependent on the amount of perpendicular light flux falling on its surface, and a tracking system tries to parallel the vector area of the solar panel surface to the incident solar flux. We present a tracking technique based on a two-axis sun sensor which can be used to increase the power output from a number of CPV arrays connected together in a solar power plant. The outdoor testing procedure of the developed two-axis sun sensor is discussed. The detail of the algorithm used together with the related sun tracking equipment is also presented and discussed for the new two axes sun tracking system.

14 SOLAR ENERGY↗

Knowledge gaps for neuromorphic ionic computing

BACKGROUND Neuromorphic computing, inspired by the human brain’s ability to process information efficiently, represents a transformative approach to computation. In this Review, we explore the emerging field of neuromorphic ionic computing, which leverages ionic conduction and coupling to mimic neural processes, and identify critical knowledge gaps that must be addressed to realize its full potential. A central theme of the discussion is energy efficiency, a challenge that is both a limitation and an opportunity for this technology. Although complementary metal-oxide semiconductor (CMOS)–based neuromorphic technologies have made strides in scaling to billions of neurons and are increasingly applied in artificial intelligence and numerical computing, they remain orders of magnitude behind the human brain in terms of connectivity and energy efficiency. Neuromorphic ionic computing promises to overcome these limitations by leveraging the distinct architectural and operational principles of the brain. Our brains achieve this energy efficiency by combining several key features: using the same network elements to store and process information; using an incredibly complex and massively interconnected three-dimensional (3D) network of locally active elements that enables sparsity, robustness in the presence of noise, adaptation, and life-long learning; computing at comparatively low voltage and frequency; and last, taking advantage of a plethora of ions and small molecules as information carriers. Here, we propose that ionic computing systems can take advantage of similar features to achieve substantial gains in energy efficiency. ADVANCES Since the first reports of neuromorphic ionic behavior in nanofluidic channels, we have witnessed an explosion of reports that used ionic devices to produce synaptomimetic behaviors. However, achieving the goals of ionic computing requires not only implementation of much more sophisticated device functionality but also overcoming fundamental barriers in materials science, device architecture, and system integration. Current ionic devices, even those incorporating state-of-the-art materials, still suffer from limited functionality and stability, which restrict their performance and increase energy demands. Developing new materials with enhanced ionic properties is essential to overcome these limitations. Similarly, the design of neuromorphic devices must evolve to leverage the particular advantages of ionic processes. Existing architectures often follow a single-information-carrier logic of conventional electronics or are constructed of mesoscale fluidics, failing to capitalize on the energy-efficient mechanisms inherent to ionic systems or implement the multiple-information-carrier paradigm. Current neuromorphic chips focus on large-scale networks of analog memory elements based on mechanisms such as charge trap (flash), filamentary, phase change, or spin, which are built on top of a network of artificial CMOS neurons. Although such prototype networks have achieved impressive performance, it is difficult to envision how they can implement the key features such as massive connectivity, sophisticated plasticity, adaptability, sparsity, and “multichromatic” computing. Although small-scale devices have demonstrated promising results, integrating them, maintaining energy efficiency, and implementing temperature control as systems grow in complexity and size to computationally relevant scale remain major hurdles. Furthermore, interfacing neuromorphic ionic devices with existing computing technologies presents technical and conceptual challenges that will require innovative approaches that combine insights from neuroscience, materials science, and engineering. OUTLOOK Despite these challenges, the potential impact of neuromorphic ionic computing is profound with potential applications ranging from artificial intelligence to robotics and beyond. We also argue that neuromorphic ionic computing systems should not, at least in the beginning, compete with CMOS technologies but rather should focus on applications that require extreme energy efficiency with chemical and/or biological compatibility, such as biomedical applications (for example, brain-computer interfaces), environmental monitoring, and agricultural and food applications. Ultimately, this Review highlights the crucial role of interdisciplinary collaboration in advancing the field. Neuromorphic ionic computing is not merely a technological innovation; it represents a substantial step toward sustainable computation, aligning with the growing demand for energy-conscious solutions in a world that is increasingly reliant on data and computation.

Neuromorphic↗

Self-Navigating Robotic Assistant for Long-term Wide Area Floor Contamination Monitoring - 18213

For Routine Contamination Testing (RCT) of an operational nuclear facility or one undergoing Decontamination and Decommissioning (D and D), workers utilize specified travel corridors (enclosed hallways or areas designated with tape or rope in open spaces) to minimize the risk of spreading contamination. Radiation health technicians must continually monitor these areas for contamination to minimize worker dose and prevent its spread to uncontrolled areas. For many sites, it is impractical for the technician to adequately survey such areas. The Nuclear Robotics Group (NRG) at UT Austin has developed a self-navigating robotic platform equipped with radiation detectors to augment the technician to improve both survey coverage and frequency. While maintaining a controlled velocity and floor-to-sensor distance, the system takes radiation measurements to identify and/or localize contamination sources. Readings above background levels are reported to a remote workstation to allow technicians to then respond accordingly. The completion of these mundane tasks by a robot allows the trained technicians to focus instead on operational tasks, D and D, or responding to alarms. The robot generates a map using a Lidar range sensor with established Simultaneous Localization and Mapping (SLAM) techniques. Range data is then compared to previously generated maps for robot localization and autonomous navigation. To maximize coverage, the robot executes a Complete Coverage Path Planning (CCPP) algorithm using a static map that is discretized into cells to assure comprehensive yet efficient coverage. As the system executes the generated plan, it builds a radiation map of the area for the technician to validate and verify the results. If contamination is detected, the robot will minimize the spread of contamination by stopping, sounding an alarm, and alerting the remote technician. The proposed system allows for long-term deployment with minimal human intervention. It is robust to both highly dynamic obstacles (i.e. personnel) and low dynamic changes (i.e. temporally stored cart) in the environment. This is accomplished through continuous spatiotemporal updating of the static map, and the utilization of a supervisory controller that allows the system to reliably respond to unexpected events or states (i.e. low battery, closed door, etc.). While the supervisory controller can respond to temporary complications, the system also recognizes the environment can change slowly over time necessitating that the CCPP automatically be updated to account for observed 'permanent' changes to the environment. To do this, the robot uses intermediate probabilistic data to make intelligent planning decisions offline while still maintaining local reactive obstacle avoidance and safety features. The self-updating of the map allows the robot to run longer without the need to remake the map, leading to more consistent monitoring data and less maintenance from the human technician. This paper will review the current system hardware, the CCPP algorithm, the supervisory controller, the probabilistic algorithm for identifying permanent changes, and recent experiments at UT Austin to experimentally validate the reported sensor coverage. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

A Multidimensional Holistic Framework for the Security of Distributed Energy and Control Systems

The digitization of smart grid distributed generation and industrial control systems has prompted utilities to deploy tools with ubiquitous communications that potentially widen the attack surface. The utilities still continue to rely on the traditional cybersecurity technologies, such as firewalls, anti-malware tools, and passwords that do not ensure security across all dimensions of the information assurance model required for a strong cybersecurity business process. This paper proposes a multidimensional holistic framework that addresses this gap through advanced technologies, intelligent algorithms, and continued assessments. To show proof, the layered defense model, a solution dimension of the framework, is integrated into the National Renewable Energy Laboratory's Security and Resilience Testbed to replicate a utility's enterprise and substation networks. The model is used to evaluate the security and resilience of microgrid control systems, and, based on the insights gathered, recommend best practices for utility cybersecurity analysts for a strong business process. Here, with this baseline, the paper conceptually introduces intelligence-driven solutions comprising contextual data analysis and machine learning to respond to advanced persistent threats sponsored by campaign efforts.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Trusted Simulation: Considering Model Quality in the Context of User Trust

A high‐quality simulation model should help its users to easily and appropriately calibrate their trust in the model. Traditional evaluation metrics such as validation and robustness are necessary but insufficient for this task. Trust calibration depends on factors like the model's transparency, applicability to intended use, usability, reputation, and consideration of potential bias. This article proposes a framework for designing and evaluating system dynamics models by considering factors that contribute to the proper calibration of user trust. This framework takes inspiration from trusted artificial intelligence, broadening our traditional concept of model quality and explicitly focusing on what users need to consider a model trustworthy and to understand the model's relevance to its intended purpose. The trusted simulation framework can improve our integration of model quality activities throughout the modeling process, leading to more impactful and better‐targeted model design, development, and evaluation.

Naugle, Asmeret Bier [Sandia National Laboratories↗

Explaining and predicting human behavior and social dynamics in simulated virtual worlds: reproducibility, generalizability, and robustness of causal discovery methods

Ground Truth program was designed to evaluate social science modeling approaches using simulation test beds with ground truth intentionally and systematically embedded to understand and model complex Human Domain systems and their dynamics Lazer et al. (Science 369:1060–1062, 2020). Our multidisciplinary team of data scientists, statisticians, experts in Artificial Intelligence (AI) and visual analytics had a unique role on the program to investigate accuracy, reproducibility, generalizability, and robustness of the state-of-the-art (SOTA) causal structure learning approaches applied to fully observed and sampled simulated data across virtual worlds. In addition, we analyzed the feasibility of using machine learning models to predict future social behavior with and without causal knowledge explicitly embedded. In this paper, we first present our causal modeling approach to discover the causal structure of four virtual worlds produced by the simulation teams—Urban Life, Financial Governance, Disaster and Geopolitical Conflict. Our approach adapts the state-of-the-art causal discovery (including ensemble models), machine learning, data analytics, and visualization techniques to allow a human-machine team to reverse-engineer the true causal relations from sampled and fully observed data. We next present our reproducibility analysis of two research methods team’s performance using a range of causal discovery models applied to both sampled and fully observed data, and analyze their effectiveness and limitations. We further investigate the generalizability and robustness to sampling of the SOTA causal discovery approaches on additional simulated datasets with known ground truth. Our results reveal the limitations of existing causal modeling approaches when applied to large-scale, noisy, high-dimensional data with unobserved variables and unknown relationships between them. We show that the SOTA causal models explored in our experiments are not designed to take advantage from vasts amounts of data and have difficulty recovering ground truth when latent confounders are present; they do not generalize well across simulation scenarios and are not robust to sampling; they are vulnerable to data and modeling assumptions, and therefore, the results are hard to reproduce. Finally, when we outline lessons learned and provide recommendations to improve models for causal discovery and prediction of human social behavior from observational data, we highlight the importance of learning data to knowledge representations or transformations to improve causal discovery and describe the benefit of causal feature selection for predictive and prescriptive modeling.

97 MATHEMATICS AND COMPUTING↗