TS-NET: COMBINING MODALITY SPECIFIC AND COMMON FEATURES FOR MULTIMODAL PATCH MATCHING

Sovann En 1 Alexis Lechervy 1 Frédéric Jurie 1
1 Equipe Image - Laboratoire GREYC - UMR6072
GREYC - Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen
Abstract : Multimodal patch matching addresses the problem of finding the correspondences between image patches from two different modalities, e.g. RGB vs sketch or RGB vs near-infrared. The comparison of patches of different modalities can be done by discovering the information common to both modalities (Siamese like approaches) or the modality-specific information (Pseudo-Siamese like approaches). We observed that none of these two scenarios is optimal. This motivates us to propose a three-stream architecture, dubbed as TS-Net, combining the benefits of the two. In addition, we show that adding extra constraints in the intermediate layers of such networks further boosts the performance. Experimentations on three multimodal datasets show significant performance gains in comparison with Siamese and Pseudo-Siamese networks.
Type de document :
Communication dans un congrès
ICIP, Oct 2018, Athens, Greece
Liste complète des métadonnées

Littérature citée [20 références]  Voir  Masquer  Télécharger

https://hal.archives-ouvertes.fr/hal-01804551
Contributeur : Frederic Jurie <>
Soumis le : jeudi 31 mai 2018 - 17:14:17
Dernière modification le : jeudi 7 février 2019 - 14:46:44
Document(s) archivé(s) le : samedi 1 septembre 2018 - 14:56:04

Fichiers

main.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01804551, version 1
  • ARXIV : 1806.01550

Citation

Sovann En, Alexis Lechervy, Frédéric Jurie. TS-NET: COMBINING MODALITY SPECIFIC AND COMMON FEATURES FOR MULTIMODAL PATCH MATCHING. ICIP, Oct 2018, Athens, Greece. 〈hal-01804551〉

Partager

Métriques

Consultations de la notice

90

Téléchargements de fichiers

66