Lifelog Semantic Annotation using deep visual features and metadata-derived descriptors

Bahjat Safadi (1, 2, 3) , Philippe Mulhem (1, 2, 3) , Georges Quénot (1, 2, 3) , Jean-Pierre Chevallet (1, 2, 3)

1 LIG - Laboratoire d'Informatique de Grenoble
2 UGA [2016-2019] - Université Grenoble Alpes [2016-2019]
3 MRIM - Modélisation et Recherche d’Information Multimédia [Grenoble]

This paper describes a method for querying lifelog data from visual content and from metadata associated with the recorded images. Our approach mainly relies on mapping the query terms to visual concepts computed on the Lifelogs images according to two separated learning schemes based on use of deep visual features. A post-processing is then performed if the topic is related to time, location or activity information associated with the images. This work was evaluated in the context of the Lifelog Semantic Access sub-task of the NTCIR-12 (2016). The results obtained are promising for a first participation to such a task, with an event-based MAP above 29% and an event-based nDCG value close to 39%.

Domaines: Recherche d'information [cs.IR]

Bahjat Safadi, Philippe Mulhem, Georges Quénot, Jean-Pierre Chevallet. Lifelog Semantic Annotation using deep visual features and metadata-derived descriptors. 14th International Workshop on Content-Based Multimedia Indexing (CBMI), Jun 2016, Bucarest, Romania. pp.1 - 6, ⟨10.1109/CBMI.2016.7500247⟩. ⟨hal-01572659⟩