Unsupervised Visual and Textual Information Fusion in CBMIR Using Graph-Based Methods

Julien Ah-Pine (1) , Gabriela Csurka (2) , Stéphane Clinchant (2)

1 ERIC - Entrepôts, Représentation et Ingénierie des Connaissances
2 Xerox Research Centre Europe [Meylan]

Multimedia collections are more than ever growing in size and diversity. Effective multimedia retrieval systems are thus critical to access these datasets from the end-user perspective and in a scalable way. We are interested in repositories of image/text multimedia objects and we study multimodal information fusion techniques in the context of content based multimedia information retrieval. We focus on graph based methods which have proven to provide state-of-the-art performances. We particularly examine two of such methods: cross-media similarities and random walk based scores. From a theoretical viewpoint, we propose a unifying graph based framework which encompasses the two aforementioned approaches. Our proposal allows us to highlight the core features one should consider when using a graph based technique for the combination of visual and textual information. We compare cross-media and random walk based results using three different real-world datasets. From a practical standpoint, our extended empirical analyses allow us to provide insights and guidelines about the use of graph based methods for multimodal information fusion in content based multimedia information retrieval.

Mots clés: Content based multimedia information retrieval; Information fusion; Graph based methods; Cross-media similarity; Random Walk; Visual reranking

Domaines: Recherche d'information [cs.IR]; Intelligence artificielle [cs.AI]

Julien Ah-Pine, Gabriela Csurka, Stéphane Clinchant. Unsupervised Visual and Textual Information Fusion in CBMIR Using Graph-Based Methods. ACM Transactions on Information Systems, 2015, 33 (2), pp.9. ⟨10.1145/2699668⟩. ⟨hal-01504636⟩