HAL will be down for maintenance from Friday, June 10 at 4pm through Monday, June 13 at 9am. More information
Skip to Main content Skip to Navigation
Conference papers

CentralNet: a Multilayer Approach for Multimodal Fusion

Valentin Vielzeuf 1, 2 Alexis Lechervy 1 Stéphane Pateux 2 Frédéric Jurie 1
1 Equipe Image - Laboratoire GREYC - UMR6072
GREYC - Groupe de Recherche en Informatique, Image et Instrumentation de Caen
Abstract : This paper proposes a novel multimodal fusion approach, aiming to produce best possible decisions by integrating information coming from multiple media. While most of the past multimodal approaches either work by projecting the features of different modalities into the same space, or by coordinating the representations of each modality through the use of constraints, our approach borrows from both visions. More specifically, assuming each modality can be processed by a separated deep convolutional network, allowing to take decisions independently from each modality, we introduce a central network linking the modality specific networks. This central network not only provides a common feature embedding but also regularizes the modality specific networks through the use of multi-task learning. The proposed approach is validated on 4 different computer vision tasks on which it consistently improves the accuracy of existing multimodal fusion approaches.
Complete list of metadata

Contributor : Valentin Vielzeuf Connect in order to contact the contributor
Submitted on : Tuesday, August 21, 2018 - 9:24:00 AM
Last modification on : Wednesday, November 3, 2021 - 5:12:40 AM
Long-term archiving on: : Thursday, November 22, 2018 - 12:49:41 PM


Files produced by the author(s)


  • HAL Id : hal-01858560, version 1
  • ARXIV : 1808.07275


Valentin Vielzeuf, Alexis Lechervy, Stéphane Pateux, Frédéric Jurie. CentralNet: a Multilayer Approach for Multimodal Fusion. European Conference on Computer Vision Workshops: Multimodal Learning and Applications, Sep 2018, Munich, Germany. ⟨hal-01858560⟩



Record views


Files downloads