On effectiveness measures and relevance functions in ranking INEX systems

Huyen-Trang Vu 1 Patrick Gallinari 1
1 MALIRE - Machine Learning and Information Retrieval
LIP6 - Laboratoire d'Informatique de Paris 6
Abstract : This paper investigates the effect of performance measures and relevance functions in comparing retrieval systems in INEX, an evaluation forum dedicated to XML retrieval. We focus on two interdependent challenges which arise when evaluating XML retrieval systems, namely weak ordering issue of retrieved lists and multivalued relevance scales. Our analysis provides empirical evidence about the reasonableness of popular assumptions in information retrieval (IR) evaluation which state that ties can be ignored and binary relevance is sufficient. We also shed light on the impact of a parameter in Q-measure [18] on the sensitivity of the metric.
Document type :
Conference papers
Complete list of metadatas

https://hal.archives-ouvertes.fr/hal-01416680
Contributor : Lip6 Publications <>
Submitted on : Wednesday, December 14, 2016 - 5:03:06 PM
Last modification on : Thursday, March 21, 2019 - 12:58:54 PM

Links full text

Identifiers

Citation

Huyen-Trang Vu, Patrick Gallinari. On effectiveness measures and relevance functions in ranking INEX systems. Asia Information Retrieval Symposium (AIRS'05), Oct 2005, Jeju, South Korea. pp.312-327, ⟨10.1007/11562382_24⟩. ⟨hal-01416680⟩

Share

Metrics

Record views

206