Model Evaluation and Hindcasting: An Experiment with an Integrated Assessment Model
Integrated assessment models have been extensively used for analyzing long term energy and greenhouse emissions trajectories and have influenced key policies on this subject. Though admittedly these models are focused on the long term trajectories, how well these models are able to capture historical dynamics is an open question. In a first experiment of its kind, we present a framework for evaluation of such integrated assessment models. We use Global Change Assessment Model for this zero order experiment, and focus on the building sector results for USA. We calibrate the model for 1990 and run it forward up to 2095 in five year time steps. This gives us results for 1995, 2000, 2005 and 2010 which we compare to observed historical data at both fuel level and service level. We focus on bringing out the key insights for the wider process of model evaluation through our experiment with GCAM. We begin with highlighting that creation of an evaluation dataset and identification of key evaluation metric is the foremost challenge in the evaluation process. Our analysis highlights that estimation of functional form of the relationship between energy service demand, which is an unobserved variable, and its drivers is a significant challenge in the absence of adequate historical data for both the dependent and driver variables. Historical data availability for key metrics is a serious limiting factor in the process of evaluation. Interestingly, service level data against which such models need to be evaluated are itself a result of models. Thus for energy services, the best we can do is compare our model results with other model results rather than observed and measured data. We show that long term models, by the nature of their construction, will most likely underestimate the rapid growth in some services observed in a short time span. Also, we learn that modeling saturated energy services like space heating is easier than modeling unsaturated services like space cooling and understanding that how far a service is from its saturation level is a key question which we probably don’t have an answer to. Finally and most importantly, even if long term models partially miss the short term dynamics, the long term insights provides by these models is fairly robust. We conclude by highlighting that our work is the first step in the much wider process of integrated assessment model evaluation and will hence have its own limitations. Future evaluation research work should build upon this zero order experiment for improving our modeling of human and coupled earth systems.