Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Program Evaluation”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

The transportation factor in site evaluations for spent nuclear fuel storage and disposal facilities - 15460

New facility siting studies will be necessary to fulfill the Blue Ribbon Commission (BRC) on America's Nuclear Future 2012 recommendation to develop consolidated interim storage (CIS) for spent nuclear fuel in the next decade. Future site evaluations for used nuclear fuel (UNF) storage and disposal facilities must address transportation requirements, especially access to mainline railroads, early in the site selection process. The U.S. Department of Energy (DOE) in 2013 proposed a strategy for managing spent nuclear fuel in accordance with the BRC recommendations by establishing a pilot interim storage facility by 2021 and a consolidated interim storage facility by 2025. The authors demonstrate that currently available geographic information systems and already adopted siting criteria (defining favorable and unfavorable conditions) allow the identification of the transportation portion of a storage facility siting process to begin without delay. Ready access to the mainline rail system is absolutely critical since existing rail access, or easily constructed access, is a highly favorable condition for successful siting. Despite widespread rail abandonments over the past three decades, vast areas of the United States retain mainline rail service or could connect to existing mainlines by building new rail spurs of less than 25-50 miles. This precondition for transportation planning is critical. Planners need to start with the beginning - availability of rail transportation to and from the interim storage facility/facilities. A second step in transportation screening would be to evaluate those potential site locations within 50 miles of existing mainline railroads using other appropriate variables. This effort could begin with the application of the 1984 repository siting guidelines for transportation (10 CFR 960.5-2-7)[1]. Repository surface facilities have essentially the same transportation requirements as storage facilities. The guidelines in 10 CFR 960 provide a readily available template for evaluating sites based on transportation risks, impacts and costs. Initial site screening, after considering proximity to mainline railroads as the precursory step, should evaluate: 1) proximity to highly populated areas; 2) proximity to interstate highways; 3) proximity to environmentally sensitive areas and lands protected by federal, state, or tribal governments; and 4) proximity to military installations, hazardous materials facilities, or other facilities that might impact transportation safety and security.Several challenges exist which prevent effective transportation planning. The diversity of canister and cask types and SNF characteristics and conditions, the increasingly limited rail access at the shipping sites and the ease with which public attention can be marshalled against a waste management program, all suggest that future efforts need to identify potential storage sites that reduce transportation impacts and costs by reducing near-site transportation construction and reduce total system transportation impacts. This can most easily be accomplished by reducing shipping distances in a process that is easily defined and implemented. The paper concludes that the DOE should develop a site screening and evaluation program with transportation as a central consideration. (authors)

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

INTER-DIGITAL CAPACITIVE SENSOR FOR EVALUATING CABLE INSULATION THROUGH JACKET

Interdigital capacitive (IDC) sensors have previously been used to measure cable-age/damage indicative dielectric properties of cable insulation when placed in direct contact with the insulation. Often cable insulation is covered by a polymer jacket precluding direct contact with the insulation. As part of a continuing program to develop and evaluate nondestructive examination methods that may be applied to cable condition assessment, an IDC to measure the effect of cable polymer dielectric change through a covering jacket has been developed. The IDC consists of two fork-like electrodes facing each other with the fork tines interspersed and separated by a small gap. On one electrode, the gap is 5 mm while on the other, the gap is 2 mm. Electric field depth-of-field penetrations are similar to the electrode gaps. The electrodes are printed on a flexible substrate that can be conformed to the outside jacketed surface of a cylindrical cable, with tines parallel to the cable axis. The electrodes are connected to a broad frequency-spectral impedance meter that senses the capacitance between the narrowly gapped electrode tines. This capacitance is known to vary as a function of the permittivity of any material in close proximity to the electrodes. For jacketed cable systems, the IDC response is dominated by the jacket but, by analyzing measurements from IDC sensors with different depths-of-field penetration into the cable under test, the influence of the cable jacket material (primarily influenced by the shallow-depth-of field IDC) can be separated from the deeper-depth-of-field IDC sensing the combined effects of both jacket and underlying insulation. No other NDE method is known that is capable of sensing insulation dielectric properties through an overlying jacket.

Cable NDE, Interdigital Capacitance, Insulation Pe↗

Evaluation of Visual Analytics Environments: The Road to the Visual Analytics Science and Technology Challenge Evaluation Methodology

The evaluation of visual analytics environments was a topic in Illuminating the Path [Thomas 2005] as a critical aspect of moving research into practice. For a thorough understanding of the utility of the systems available, evaluation not only involves assessing the visualizations, interactions or data processing algorithms themselves, but also the complex processes that a tool is meant to support (such as exploratory data analysis and reasoning, communication through visualization, or collaborative data analysis [Lam 2012; Carpendale 2007]). Researchers and practitioners in the field have long identified many of the challenges faced when planning, conducting, and executing an evaluation of a visualization tool or system [Plaisant 2004]. Evaluation is needed to verify that algorithms and software systems work correctly and that they represent improvements over the current infrastructure. Additionally to effectively transfer new software into a working environment, it is necessary to ensure that the software has utility for the end-users and that the software can be incorporated into the end-user’s infrastructure and work practices. Evaluation test beds require datasets, tasks, metrics and evaluation methodologies. As noted in [Thomas 2005] it is difficult and expensive for any one researcher to setup an evaluation test bed so in many cases evaluation is setup for communities of researchers or for various research projects or programs. Examples of successful community evaluations can be found [Chinchor 1993; Voorhees 2007; FRGC 2012]. As visual analytics environments are intended to facilitate the work of human analysts, one aspect of evaluation needs to focus on the utility of the software to the end-user. This requires representative users, representative tasks, and metrics that measure the utility to the end-user. This is even more difficult as now one aspect of the test methodology is access to representative end-users to participate in the evaluation. In many cases the sensitive nature of data and tasks and difficult access to busy analysts puts even more of a burden on researchers to complete this type of evaluation. User-centered design goes beyond evaluation and starts with the user [Beyer 1997, Shneiderman 2009]. Having some knowledge of the type of data, tasks, and work practices helps researchers and developers know the correct paths to pursue in their work. When access to the end-users is problematic at best and impossible at worst, user-centered design becomes difficult. Researchers are unlikely to go to work on the type of problems faced by inaccessible users. Commercial vendors have difficulties evaluating and improving their products when they cannot observe real users working with their products. In well-established fields such as web site design or office software design, user-interface guidelines have been developed based on the results of empirical studies or the experience of experts. Guidelines can speed up the design process and replace some of the need for observation of actual users [heuristics review references]. In 2006 when the visual analytics community was initially getting organized, no such guidelines existed. Therefore, we were faced with the problem of developing an evaluation framework for the field of visual analytics that would provide representative situations and datasets, representative tasks and utility metrics, and finally a test methodology which would include a surrogate for representative users, increase interest in conducting research in the field, and provide sufficient feedback to the researchers so that they could improve their systems.

evaluation↗

Attracting Future Radiation Oncologists: An Analysis of the National Resident Matching Program Data Trends From 2004 to 2015

Purpose: A significant physician shortage has been projected to occur by 2025, and demand for oncologists is expected to outpace supply to an even greater degree. In response to this, many have called to increase the number of radiation oncology residency positions. The purpose of this study is to evaluate National Resident Matching Program (NRMP) data for the number of residency positions between 2004 and 2015 as well as the number and caliber of applicants for those positions and to compare radiation oncology to all residency specialties. Methods: NRMP data for all specialties participating in the match, including radiation oncology, were assessed over time examining the number of programs participating in the match, the number of positions offered, and the ratio of applicants to positions in the match from 2004 to 2015. Results: From 2004 to 2015, the number of total programs participating in the match has increased by 26.7%, compared to the increase of 28.6% in the number of radiation oncology programs from during the same time period. The total number of positions offered in the match increased by 53.4%, whereas radiation oncology positions increased by 56.3%, during the same time period. The ratio of applicants (defined as those selecting a specialty as their first or only choice) to positions for all specialties has fluctuated over this time period and has gone from 1.21 to 1.15, whereas radiation oncology experienced a decrease from 1.45 to 1.14. Conclusions: NRMP data suggest that senior medical student applications to radiation oncology are decreasing compared to those of other specialties. If we hope to continue to attract the best and brightest to enter our field, we must continue to support early exposure to radiation oncology, positive educational experiences, and dedicated mentorship to interested medical students.

62 RADIOLOGY AND NUCLEAR MEDICINE↗

Study of Multi-Scale Cloud Processes Over the Tropical Western Pacific Using Cloud-Resolving Models Constrained by Satellite Data

Clouds in the tropical western Pacific are an integral part of the large scale environment. An improved understanding of the multi-scale structure of clouds and their interactions with the environment is critical to the ARM (Atmospheric Radiation Measurement) program for developing and evaluating cloud parameterizations, understanding the consequences of model biases, and providing a context for interpreting the observational data collected over the ARM Tropical Western Pacific (TWP) sites. Three-dimensional cloud resolving models (CRMs) are powerful tools for developing and evaluating cloud parameterizations. However, a significant challenge in using CRMs in the TWP is that the region lacks conventional data, so large uncertainty exists in defining the large-scale environment for clouds. This project links several aspects of the ARM program, from measurements to providing improved analyses, and from cloud-resolving modeling to climate-scale modeling and parameterization development, with the overall objective to improve the representations of clouds in climate models and to simulate and quantify resolved cloud effects on the large-scale environment. Our objectives will be achieved through a series of tasks focusing on the use of the Weather Research and Forecasting (WRF) model and ARM data. Our approach includes: - Perform assimilation of COSMIC GPS radio occultation and other satellites products using the WRF Ensemble Kalman Filter assimilation system to represent the tropical large-scale environment at 36 km grid resolution. This high-resolution analysis can be used by the community to derive forcing products for single-column models or cloud-resolving models. - Perform cloud-resolving simulations using WRF and its nesting capabilities, driven by the improved regional analysis and evaluate the simulations against ARM datasets such as from TWP-ICE to optimize the microphysics parameters for this region. A cirrus study (Mace and co-authors) already exists for TWP-ICE using satellite and ground-based observations. - Perform numerical experiments using WRF to investigate how convection over tropical islands in the Maritime Continent interacts with large-scale circulation and affects convection in nearby regions. - Evaluate and apply WRF as a testbed for GCM cloud parameterizations, utilizing the ability of WRF to run on multiple scales (from cloud resolving to global) to isolate resolution and physics issues from dynamical and model framework issues. Key products will be disseminated to the ARM and larger community through distribution of data archives, including model outputs from the data assimilation products and cloud resolving simulations, and publications.

54 ENVIRONMENTAL SCIENCES↗

Evaluation of the US Department of Energy Weatherization Innovation Pilot Program (2010-2014)

This report contains results from analysis conducted on each of the Weatherization Innovation Pilot Program (WIPP) grants awarded to 16 organizations by the US Department of Energy (DOE) in 2010. The purpose of WIPP was to explore the potential adoptability or replicability of innovative processes or technologies for the enhancement of DOE’s Weatherization Assistance Program (WAP). DOE initiated the WIPP grant to accelerate effective innovations in home energy efficiency and other WAP mission-related goals for income-qualifying households of low socioeconomic status. This study was performed alongside a broader, national evaluation of WAP conducted by Oak Ridge National Laboratory (ORNL) for DOE.

54 ENVIRONMENTAL SCIENCES↗

Fracture Toughness Evaluation for Sandia Mock-up Stainless Steel Canister Weldment Using Spiral Notch Torsion Fracture Toughness Test

Many radioactive materials within the nuclear fuel cycle present a significant hazard. Such materials include spent nuclear fuel, high-level waste, legacy waste and other nuclear materials that have no current outlet than storage. These materials are often held in long-term storage as an interim stage within their lifecycle. Lifecycles can include reuse or disposal. Safety and security of spent nuclear fuel (SNF) interim storage installations are very important, due to a great concentration of fission products, actinides and activation products. Stress corrosion cracking of interim storage containers has been identified as a high priority data gap by the Department of Energy. However, little has been done with regards to the canister weld material properties, including the fracture toughness, and their impact on stress corrosion, until recently from Sandia National Laboratory canister mockup program. Furthermore, because no post-weld heat treatment was required, the associated high tensile residual stress within these canister welds can drive the initiation and growth of stress corrosion cracking (SCC) cracks. Thus, the fracture mechanical property, such as fracture toughness, is an essential information to understand the canister weldment fracture resistance behavior and to support mitigation protocol development to counter SCC. In this work, we carried out fracture toughness evaluation on the received mock-up canister weldments from Sandia, using spiral notch torsion fracture test technology. The details of the weldment fracture toughness evaluation program development at ORNL are presented in this progress report. The objective of this research is to collect fracture toughness data on the mock-up canister weldment from Sandia National Laboratory. Data will be collected under quasi-static fracture loading using the Oak Ridge National Laboratory (ORNL) developed Spiral Notch Torsion Test (SNTT) technology carried out on a biaxial tension/torsion tester. These data will be used to support the ongoing SNF dry storage canister modeling activities and to address regulatory issues associated with canister container integrity and the follow-on SNF system transport reliability. SNTT has been a recent breakthrough in measuring fracture toughness for different materials, including metals, ceramics, concrete and polymers composites. Due to its high geometry constraint and unique loading condition, SNTT can be used to measure the fracture toughness with smaller specimens without concern of size effects. The application of SNTT to brittle materials has been proved to be successful. The micro-cracks induced by original notches in brittle materials could ensure crack growth in SNTT samples. Therefore, no fatigue pre-cracks are needed. The application of SNTT to the ductile material to generate valid toughness data will require a test sample with sufficient crack length to increase the sample fracture tip localized constraint. Fatigue pre-crack growth techniques are employed to introduce sharp crack front into the sample. Previously, only rough calculations were applied to estimate the compliance evolution in the SNTT crack growth process; while accurate quantitative descriptions have never been attempted. This generates an urgent need to understand the crack evolution during the SNTT fracture testing process of ductile materials. The newly developed governing equations for SNTT crack growth estimate are used to control the fatigue crack growth to effectively reaching the target crack length. The detailed SNTT approach and its estimated fracture toughness for the as-received Sandia canister weldment and the associated baseline 304 stainless steel are presented in this report. In the past SNTT technology has also been successfully applied to investigate the fracture behavior of X52 and X80 steels and the X52 welded materials used for hydrogen infrastructures. The current study in supporting Sandia SNF Dry Storage Mock-up Canister Program is the first attempt of applying SNTT to the highly ductile stainless steel and the associated weld materials. The SNTT test results indicate that SNTT method is a reliable test approach with good repeatability in applying to SS304 steel and SS304/308 weld material. The estimate JQ upon fracture for the baseline SS304 steel is at 1618 lb/in. The estimate JQ upon fracture for the SS304/308 weld from HAZ and weld specimens are 720 lb/in and 850 lb/in, respectively.

12 MANAGEMENT OF RADIOACTIVE AND NON-RADIOACTIVE W↗

Advanced Reactors Spent Fuel & Waste Science and Technology Program

Based on the higher interest in Advanced Reactor (AR) deployment (e.g., ARDP ) for potential new fuel cycles, the Spent Fuel & Waste Science and Technology (SFWST) Program has begun to evaluate the possible implications of long-term management and final disposition of potential Advance Reactor spent nuclear fuels (SNF) that would be generated in potential advanced reactors. Safely managing and dispositioning the potential future AR SNF, and any other associated radioactive wastes, is the primary focus of this initial preliminary assessment of those. This paper summarizes three primary tasks the Spent Fuel & Waste Science and Technology (SFWST) Program is executing (or collaborating on) related to the back end of the nuclear fuel cycle (BENFC) for potential future advanced reactors: 1. Advanced Reactors Spent Fuel and Waste Streams: Strategies for the BENFC This set of activities define a high-level strategy for how to systematically approach, identify, and close research and development (R&D) gaps/potential issues associated with long-term management and final disposition of AR SNF and other possible AR waste streams. This task involves summarizing advanced reactor concepts, their likely SNF and other waste forms, and identifying previous experience with similar materials, for example from DOE-managed SNF, with closely related characteristics to the potential future AR SNF. Technical R&D gaps between the breadth of detailed understanding for safe storage, transportation and disposal of the existing light water reactor SNF fuel cycle (e.g., see NASEM, 2022) and potential future fuel cycles based on advanced reactors would then be identified. 2. Characterization and Packaging Options of Advanced Reactor SNF These activities evaluate characteristics and packaging options for advanced reactor spent fuel forms. The fuel forms are categorized into three types: (1) tri-structural isotropic (TRISO), (2) metallic, and (3) fuel salt. Emphasis is given to TRISO and metallic SNF and additional waste streams from such AR as driven by the near-term anticipated operation of the Xe-100 and the Natrium reactors as advanced-reactor demonstrations1. Preliminary information for the spent-fuel salt discharged from molten-salt reactors (MSRs) is examined to provide a baseline for future efforts. All calculations and assumptions in this work are based on publicly available information. The following characteristics are calculated or estimated for use in the preliminary assessments: SNF volume and mass, radiation/activity levels through time, thermal conditions through time, potential radionuclide source terms, chemical interactions and evolutions, disposal inventories, and waste-form lifetime. Using those characteristics, calculations to determine the applicability of existing canister designs were performed. These evaluations included geometric (e.g., dimension, volume) and mass/weight considerations, known operational approaches and loading procedures, physical and chemical considerations/conditions for storage environments, as-loaded radiation, thermal, and criticality analyses to identify constraints for storage, transportation, and disposal. 3. Back-End Management of Advanced Reactors (BEMAR) The DOE NE-8 organization has defined an Integrated Project Team to evaluate the Back End Management of Advanced Reactors (BEMAR) (includes DOE staff from a range of organizations (e.g., NE-81, NE-82, OCED) and national laboratory technical staff within the DOE NE-81 and NE-82 programs). This BEMAR group works directly with advance reactors developers to assess for the DOE the technical feasibility of storage, transportation, and disposal of AR SNF based on the characteristics provided by the developers to DOE (much of which is proprietary). The BEMAR is also tasked to develop rough-order-of-magnitude cost estimates to compare the waste management system for individual advanced reactors to existing light-water reactor management practices. To accomplish this, the BEMAR group is implementing a Systems Engineering approach.

11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS↗

Integrating Waste Heat from CO 2 Removal and Coal-Fired Flue Gas to Increase Plant Efficiency

In project DE-FE0007525, Southern Company Services demonstrated heat integration methods for the capture and sequestration of carbon dioxide produced from pulverized coal combustion. A waste heat recovery technology (termed High Efficiency System) from Mitsubishi Heavy Industries America was integrated into an existing 25-MW amine-based CO 2 capture process (Kansai Mitsubishi Carbon Dioxide Recovery Process®1) at Southern Company’s Plant Barry to evaluate improvements in the energy performance of the pulverized coal plant and CO 2 capture process. The heat integration system consists of two primary pieces of equipment: (1) the CO 2 Cooler which uses product CO 2 gas from the capture process to heat boiler condensate, and (2) the Flue Gas Cooler which uses air heater outlet flue gas to further heat boiler condensate. Both pieces of equipment were included in the pilot system. The pilot CO 2 Cooler used waste heat from the 25-MW CO 2 capture plant (but not always from product CO 2 gas, as intended). The pilot Flue Gas Cooler used heat from a slipstream of flue gas taken from downstream of Plant Barry’s air heater. The pilot also included a 0.25-MW electrostatic precipitator. The 25-MW High Efficiency System operated for approximately six weeks over a four month time period in conjunction with the 25-MW CO 2 capture facility at Plant Barry. Results from the program were used to evaluate the technical and economic feasibility of full-scale implementation of this technology. The test program quantified energy efficiency improvements to a host power plant that could be realized due to the High Efficiency System. Through the execution of this project, the team verified the integrated operation of the High Efficiency System and Kansai Mitsubishi Carbon Dioxide Recovery Process®. The ancillary benefits of the High Efficiency System were also quantified, including reduced water consumption, a decrease in toxic air emissions, and better overall air quality control systems performance.

20 FOSSIL-FUELED POWER PLANTS↗

2017 Project Peer Review: U.S. Department of Energy Wind Energy Technologies Office: Volume I

This report details the results of the 2017 Peer Review for the U.S. Department of Energy’s Office of Energy Efficiency and Renewable Energy Wind Energy Technologies Office (WETO). The purpose of the review was to evaluate projects funded by DOE from fiscal year 2014 through fiscal year 2016 for their contribution to the mission and goals of the office, assess progress against stated objectives, and appraise WETO’s overall management and performance. This volume (Volume I) includes Sections 1–5 of the report: the executive summary, synopses of the program and project-level evaluation results, and WETO’s response to the 2017 Peer Review findings. Volume II includes Sections 6 and 7—the complete program-level and project-level evaluation results—as well as the report appendices.

17 WIND ENERGY↗

Ampaire ARPA-e Electric Flight Testbed

A hybrid-electric aircraft flying testbed was developed in this program with the intent to serve as a dedicated, enduring testbed to test and evaluate ARPA-e CIRCUITS Program and other electrified aviation technologies in relevant flight environments. This testbed enabled rapid development cycles of novel and innovative technologies in the electrified aviation space, maturing them from a research lab environment to flying in an aircraft. By providing research groups with the means to test their transformative technologies in a real-world, aircraft environment, the path to validating the safety and reliability of their technologies for future commercial opportunities was greatly accelerated. Three core technologies were integrated and tested: an inverter/motor drive built by the University of Arkansas, a solid-state circuit breaker (iBreaker) built by the Illinois Institute of Technology, and a Flying Capacitor Multi-level (FCML) DC/DC converter built by the University of California, Berkeley. In each of these cases, the requirements established for safety of flight resulted in a holistic approach to the designs, evoking a deeper understanding of the potential failure modes and mitigations necessary to build a robust and flightworthy system. Further, the integration into a hybrid-electric aircraft de-risked the potential electrical and mechanical issues that cannot easily be experienced or replicated in a lab environment. The experiments were also required to undergo representative temperature, shock, and vibration testing as the FAA prescribes for this category of aircraft, facilitating familiarity with the relevant design and test guidelines necessary to commercialize the technologies. This testbed unlocks the massive potential of core power electronics technologies necessary for a safe, robust, and efficient electric aviation future. With quick iterative design, test, and flight cycles, these core technologies are on a quicker path to technology readiness level maturity and commercialization, enabling a more sustainable future for the aviation industry.

24 POWER TRANSMISSION AND DISTRIBUTION↗

Technical Feasibility Aspects of the Geothermal Resource Reporting Methodology (GRRM)

This paper reviews the technical assessment of the Geothermal Research Reporting Methodology (GRRM, http://en.openei.org/wiki/GRRM) being developed for reporting geothermal resources and project progress. The goal of the methodology is to provide the U.S. Department of Energy's Geothermal Technologies Office (GTO) with a consistent and comprehensible means of evaluating the impacts of its funding programs. The GRRM is designed to provide uniform assessment criteria for geothermal resource grades and developmental phases of geothermal resource exploration and development. This resource grade system provides information on twelve attributes of geothermal resource locations (e.g., temperature, permeability, land access) to indicate potential for geothermal development. The GTO plans to use these Protocols to help quantitatively identify the greatest barriers to geothermal development, develop measureable program goals that will have the greatest impact to geothermal deployment, objectively evaluate proposals based (in part) on a project's ability to contribute to program goals, monitor project progress, and report on GTO portfolio performance. The GRRM assesses three areas of geothermal potential: geological, socio-economic, and technical. Previous work and publications have discussed the work done on the geological aspects of this methodology (Young et al. 2015c); this paper details the development of the technical assessment of the GRRM. Technical development attributes considered include: reservoir management, drilling, logistics, and power conversion.

15 GEOTHERMAL ENERGY↗

Explaining and predicting human behavior and social dynamics in simulated virtual worlds: reproducibility, generalizability, and robustness of causal discovery methods

Ground Truth program was designed to evaluate social science modeling approaches using simulation test beds with ground truth intentionally and systematically embedded to understand and model complex Human Domain systems and their dynamics Lazer et al. (Science 369:1060–1062, 2020). Our multidisciplinary team of data scientists, statisticians, experts in Artificial Intelligence (AI) and visual analytics had a unique role on the program to investigate accuracy, reproducibility, generalizability, and robustness of the state-of-the-art (SOTA) causal structure learning approaches applied to fully observed and sampled simulated data across virtual worlds. In addition, we analyzed the feasibility of using machine learning models to predict future social behavior with and without causal knowledge explicitly embedded. In this paper, we first present our causal modeling approach to discover the causal structure of four virtual worlds produced by the simulation teams—Urban Life, Financial Governance, Disaster and Geopolitical Conflict. Our approach adapts the state-of-the-art causal discovery (including ensemble models), machine learning, data analytics, and visualization techniques to allow a human-machine team to reverse-engineer the true causal relations from sampled and fully observed data. We next present our reproducibility analysis of two research methods team’s performance using a range of causal discovery models applied to both sampled and fully observed data, and analyze their effectiveness and limitations. We further investigate the generalizability and robustness to sampling of the SOTA causal discovery approaches on additional simulated datasets with known ground truth. Our results reveal the limitations of existing causal modeling approaches when applied to large-scale, noisy, high-dimensional data with unobserved variables and unknown relationships between them. We show that the SOTA causal models explored in our experiments are not designed to take advantage from vasts amounts of data and have difficulty recovering ground truth when latent confounders are present; they do not generalize well across simulation scenarios and are not robust to sampling; they are vulnerable to data and modeling assumptions, and therefore, the results are hard to reproduce. Finally, when we outline lessons learned and provide recommendations to improve models for causal discovery and prediction of human social behavior from observational data, we highlight the importance of learning data to knowledge representations or transformations to improve causal discovery and describe the benefit of causal feature selection for predictive and prescriptive modeling.

97 MATHEMATICS AND COMPUTING↗

Measuring impacts of California agri-environmental programs using field-scale satellite data

In the past decade, California has invested over $\$$200 million in direct grants to growers to support the adoption of agricultural practices that save water and/or improve soil health while also reducing greenhouse gas emissions. Ex-post evaluation of agri-environmental outcomes of these grant programs, however, is limited. We use satellite data to monitor changes in field-level consumptive water use and greenness (i.e. normalized difference vegetation index), a proxy for agricultural productivity, for the most frequently funded crop-types (almonds, grapes, and walnuts) in two California Department of Food and Agriculture programs. Nearly 600 fields receiving funding during the 2014–2022 period were analyzed using two causal inference methods. Fields that received grants to both upgrade irrigation systems and install irrigation water management sensors showed reduced consumptive water use and greenness by an average of 3.5% and 4.2%, respectively (significant at the 10% level). In contrast, we find that the adoption of only irrigation water management sensors, which are designed to inform irrigation scheduling and management, resulted in an average increase of 4.1% and 4.8% in consumptive water use and greenness respectively (significant at the 5% level). We find negligible effects for either consumptive water use or greenness when both pump efficiency upgrades and sensors were implemented. We further find that grants for compost addition and cover cropping led to small greenness increases of 1.7% and 2.8% respectively (significant at the 10% level) and had insignificant effects on consumptive water use. Our analysis of five agri-environmental program interventions reveals that several practice outcomes may be at odds with stated program goals of reducing water use while maintaining or improving agricultural productivity.

agriculture↗