CentralNet: a Multilayer Approach for Multimodal Fusion

Valentin Vielzeuf 1, 2 Alexis Lechervy 1 Stéphane Pateux 2 Frédéric Jurie 1
1 Equipe Image - Laboratoire GREYC - UMR6072
GREYC - Groupe de Recherche en Informatique, Image, Automatique et Instrumentation de Caen
Abstract : This paper proposes a novel multimodal fusion approach, aiming to produce best possible decisions by integrating information coming from multiple media. While most of the past multimodal approaches either work by projecting the features of different modalities into the same space, or by coordinating the representations of each modality through the use of constraints, our approach borrows from both visions. More specifically, assuming each modality can be processed by a separated deep convolutional network, allowing to take decisions independently from each modality, we introduce a central network linking the modality specific networks. This central network not only provides a common feature embedding but also regularizes the modality specific networks through the use of multi-task learning. The proposed approach is validated on 4 different computer vision tasks on which it consistently improves the accuracy of existing multimodal fusion approaches.
Type de document :
Communication dans un congrès
European Conference on Computer Vision Workshops: Multimodal Learning and Applications, Sep 2018, Munich, Germany. 〈https://mula2018.github.io/〉
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-01858560
Contributeur : Valentin Vielzeuf <>
Soumis le : mardi 21 août 2018 - 09:24:00
Dernière modification le : jeudi 7 février 2019 - 17:46:42
Document(s) archivé(s) le : jeudi 22 novembre 2018 - 12:49:41

Fichiers

eccv2018submission.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-01858560, version 1
  • ARXIV : 1808.07275

Citation

Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, Frédéric Jurie. CentralNet: a Multilayer Approach for Multimodal Fusion. European Conference on Computer Vision Workshops: Multimodal Learning and Applications, Sep 2018, Munich, Germany. 〈https://mula2018.github.io/〉. 〈hal-01858560〉

Partager

Métriques

Consultations de la notice

115

Téléchargements de fichiers

384