Weakly-supervised learning of visual relations

Julia Peyre 1, 2 Ivan Laptev 1, 2 Cordelia Schmid 3 Josef Sivic 1, 2, 4
1 WILLOW - Models of visual object recognition and scene understanding
Inria de Paris, DI-ENS - Département d'informatique de l'École normale supérieure
3 Thoth - Apprentissage de modèles à partir de données massives
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann
Abstract : This paper introduces a novel approach for modeling visual relations between pairs of objects. We call relation a triplet of the form (subject, predicate, object) where the predicate is typically a preposition (eg. 'under', 'in front of') or a verb ('hold', 'ride') that links a pair of objects (subject, object). Learning such relations is challenging as the objects have different spatial configurations and appearances depending on the relation in which they occur. Another major challenge comes from the difficulty to get annotations , especially at box-level, for all possible triplets, which makes both learning and evaluation difficult. The contributions of this paper are threefold. First, we design strong yet flexible visual features that encode the appearance and spatial configuration for pairs of objects. Second, we propose a weakly-supervised discriminative clustering model to learn relations from image-level labels only. Third we introduce a new challenging dataset of unusual relations (UnRel) together with an exhaustive annotation, that enables accurate evaluation of visual relation retrieval. We show experimentally that our model results in state-of-the-art results on the visual relationship dataset [31] significantly improving performance on previously unseen relations (zero-shot learning), and confirm this observation on our newly introduced UnRel dataset.
Type de document :
Communication dans un congrès
ICCV 2017- International Conference on Computer Vision 2017, Oct 2017, Venice, Italy. ICCV 2017, International Conference on Computer Vision 2017, 2017, <http://iccv2017.thecvf.com/>
Liste complète des métadonnées


https://hal.archives-ouvertes.fr/hal-01576035
Contributeur : Julia Peyre <>
Soumis le : mardi 22 août 2017 - 15:35:37
Dernière modification le : mardi 5 septembre 2017 - 08:29:24

Fichier

weaksup_hal.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01576035, version 1
  • ARXIV : 1707.09472

Collections

Citation

Julia Peyre, Ivan Laptev, Cordelia Schmid, Josef Sivic. Weakly-supervised learning of visual relations. ICCV 2017- International Conference on Computer Vision 2017, Oct 2017, Venice, Italy. ICCV 2017, International Conference on Computer Vision 2017, 2017, <http://iccv2017.thecvf.com/>. <hal-01576035>

Partager

Métriques

Consultations de
la notice

282

Téléchargements du document

59