Semantic combination of textual and visual information in multimedia retrieval

Stéphane Clinchant (1) , Julien Ah-Pine (2) , Gabriela Csurka (1)

1 Xerox Research Centre Europe [Meylan]
2 ERIC - Entrepôts, Représentation et Ingénierie des Connaissances

The goal of this paper is to introduce a set of techniques we call semantic combination in order to efficiently fuse text and image retrieval systems in the context of multimedia information access. These techniques emerge from the observation that image and tex-tual queries are expressed at different semantic levels and that a single image query is often ambiguous. Overall, the semantic combination techniques overcome a conceptual barrier rather than a technical one: these methods can be seen as a combination of late fusion and image reranking. Albeit simple, this approach has not been used yet. We assess the proposed techniques against late and cross-media fusion using 4 different ImageCLEF datasets. Compared to late fusion, performances significantly increase on two datasets and remain similar on the two other ones.

Mots clés: Retrieval model; Search Process; Information fusion; Multimedia retrieval

Domaines: Recherche d'information [cs.IR]; Intelligence artificielle [cs.AI]

Stéphane Clinchant, Julien Ah-Pine, Gabriela Csurka. Semantic combination of textual and visual information in multimedia retrieval. 1st ACM International Conference on Multimedia Retrieval (ICMR 2008), Apr 2011, Trente, Italy. pp.44, ⟨10.1145/1991996.1992040⟩. ⟨hal-01504613⟩