MUTAN: Multimodal Tucker Fusion for Visual Question Answering - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2017

MUTAN: Multimodal Tucker Fusion for Visual Question Answering

Résumé

Bilinear models provide an appealing framework for mixing and merging information in Visual Question Answering (VQA) tasks. They help to learn high level associations between question meaning and visual concepts in the image, but they suffer from huge dimensionality issues. We introduce MUTAN, a multimodal tensor-based Tucker decomposition to efficiently parametrize bilinear interactions between visual and textual representations. Additionally to the Tucker framework, we design a low-rank matrix-based decomposition to explicitly constrain the interaction rank. With MUTAN, we control the complexity of the merging scheme while keeping nice interpretable fusion relations. We show how the Tucker decomposition framework generalizes some of the latest VQA architectures, providing state-of-the-art results.
Fichier principal
Vignette du fichier
1705.06676.pdf (6.11 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02073637 , version 1 (26-03-2019)

Identifiants

Citer

Hedi Ben-Younes, Remi Cadene, Matthieu Cord, Nicolas Thome. MUTAN: Multimodal Tucker Fusion for Visual Question Answering. 2017 IEEE International Conference on Computer Vision (ICCV), Oct 2017, Venice, Italy. pp.2631-2639, ⟨10.1109/ICCV.2017.285⟩. ⟨hal-02073637⟩
251 Consultations
86 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More