Informativeness for Adhoc IR evaluation: a measure that prevents assessing individual documents

Abstract : Informativeness measures have been used in interactive in- formation retrieval and automatic summarization evaluation. Indeed, as opposed to adhoc retrieval, these two tasks cannot rely on the Cranfield evaluation paradigm in which retrieved documents are compared to static query relevance document lists. In this paper, we explore the use of informativeness measures to evaluate adhoc task. The advantage of the proposed evaluation framework is that it does not rely on an exhaustive reference and can be used in a changing environment in which new documents occur, and for which relevance has not been assessed. We show that the correlation between the official system ranking and the informativeness measure is specifically high for most of the TREC adhoc tracks.
Complete list of metadatas

Cited literature [8 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01567065
Contributor : Open Archive Toulouse Archive Ouverte (oatao) <>
Submitted on : Friday, July 21, 2017 - 4:08:12 PM
Last modification on : Friday, June 14, 2019 - 6:31:15 PM

File

deveaud_16883.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01567065, version 1
  • OATAO : 16883

Citation

Romain Deveaud, Véronique Moriceau, Josiane Mothe, Eric San Juan. Informativeness for Adhoc IR evaluation: a measure that prevents assessing individual documents. 38th European Conference on Information Retrieval (ECIR 2016), Mar 2016, Padoue, Italy. pp. 818-823. ⟨hal-01567065⟩

Share

Metrics

Record views

205

Files downloads

109