Model Based Comparison of Discounted Cumulative Gain and Average Precision

Georges Dupret Benjamin Piwowarski 1
1 BD - Bases de Données
LIP6 - Laboratoire d'Informatique de Paris 6
Abstract : In this paper, we propose to explain Discounted Cumulative Gain (DCG) as the expectation of the total utility collected by a user given a generative probabilistic model on how users browse the result page ranking list of a search engine. We contrast this with a generalization of Average Precision, pAP, that has been defined in Dupret and Piwowarski (2010). In both cases, user decision models coupled with Web search logs allow to estimate some parameters that are usually left to the designer of a metric. In this paper, we compare the user models for DCG and pAP at the interpretation and experimental level. DCG and AP are metrics computed before a ranking function is exposed to users and as such, their role is to predict the function performance. In counterpart to prognostic metric, a diagnostic metric is computed after observing the user interactions with the result list. A commonly used diagnostic metric is the clickthrough rate at position 1, for example. In this work we show that the same user model developed for DCG can be used to derive a diagnostic version of this metric. The same hold for pAP and any metric with a proper user model. We show that not only does this diagnostic view provide new information, it also allows to define a new criterion for assessing a metric. In previous works based on user decision modeling, the performance of different metrics were compared indirectly in terms of the ability of the associated user model to predict future user actions. Here we propose a new and more direct criterion based on the ability of the prognostic version of the metric to predict the diagnostic performance.
Type de document :
Article dans une revue
Journal of Discrete Algorithms, Elsevier, 2013, 18, pp.49-62. 〈10.1016/j.jda.2012.10.002〉
Liste complète des métadonnées
Contributeur : Lip6 Publications <>
Soumis le : jeudi 20 août 2015 - 11:22:20
Dernière modification le : jeudi 13 décembre 2018 - 01:42:53

Lien texte intégral




Georges Dupret, Benjamin Piwowarski. Model Based Comparison of Discounted Cumulative Gain and Average Precision. Journal of Discrete Algorithms, Elsevier, 2013, 18, pp.49-62. 〈10.1016/j.jda.2012.10.002〉. 〈hal-01185457〉



Consultations de la notice