A farewell to the sum of Akaike weights: the benefits of alternative metrics for variable importance estimations in model selection.
Résumé
In a previous article, we advocated against using the sum of Akaike weights (SW) as a metric to distinguish between genuine and spurious variables in Information Theoretic (IT) statistical analyses. A recent article (Giam & Olden, Methods in Ecology and Evolution, 2016, 7, 388) criticises our finding and instead argues in favour of SW. It points out that (1) we performed a biased data-generation procedure and (2) we erroneously evaluated SW on its capacity to estimate the proportion of variance in the data explained by a variable. We here respond to these points.
Giam and Olden's first concern is unfounded. When using the data-generating code they proposed, SW remains very imprecise. To respond to their second concern, we first list the meanings taken by a variable's importance in the context of IT. Although, SW is presented as an estimate of variable relative importance in methodological textbooks (i.e. a variable's rank in importance or its relative contribution to the variance in the data), it is also used as a metric of variable absolute importance (i.e. a variable's absolute effect size or its statistical significance). We then compare SW to alternative metrics on its ability to estimate variable absolute or relative importance.
SW values have low repeatability across analyses. As a result, based on SW, it is hard to distinguish between variables with weak and large effects. For estimations of variable absolute importance, experimenters should prefer model-averaged parameter estimates and/or compare nested models based on evidence ratios. Sum of Akaike weights is also a poor metric of variable relative importance. We showed that correct variable ranking in importance was generally more frequent when using model-averaged standardised parameter estimates, than when using SW.
To avoid recurrent errors in ecology and evolution, we therefore warn against the use of SW for estimations of variable absolute and relative importance and we propose that experimenters should instead use model-averaged standardised parameter estimates for statistical inferences.