Direction of impact for explainable risk assessment modeling
Abstract Several graphical indicators have been recently introduced to help analysts visualize the marginal effects of inputs in complex models. The insights derived from such tools may help decision‐makers and risk analysts in designing interventions. However, we know little about the adequacy and consistency of different indicators. This work investigates popular marginal effect indicators to understand whether they yield indications consistent with the properties of the quantitative model under inspection. Specifically, we examine the notions of monotonicity, Lipschitz, and concavity consistency. Surprisingly, only PD functions satisfy all these notions of consistency. However, when selecting the indicators, in addition to consistency, analysts need to consider the risk of model extrapolation. For situations where such risk is under control, we utilize individual conditional expectations together with PD plots. Two applications, on a NASA space risk assessment model and a susceptible exposed infected recovered (SEIR) model for the COVID‐19 pandemic illustrate the insights obtained from these indicators.