Skip to Main content Skip to Navigation
Journal articles

VisQA: X-raying Vision and Language Reasoning in Transformers

Theo Jaunet 1, 2 Corentin Kervadec 2, 3 Romain Vuillemot 1 Grigory Antipov 3 Moez Baccouche 3 Christian Wolf 2 
1 SICAL - Situated Interaction, Collaboration, Adaptation and Learning
LIRIS - Laboratoire d'InfoRmatique en Image et Systèmes d'information
2 imagine - Extraction de Caractéristiques et Identification
LIRIS - Laboratoire d'InfoRmatique en Image et Systèmes d'information
Abstract : Visual Question Answering systems target answering open-ended textual questions given input images. They are a testbed for learning high-level reasoning with a primary use in HCI, for instance assistance for the visually impaired. Recent research has shown that state-of-the-art models tend to produce answers exploiting biases and shortcuts in the training data, and sometimes do not even look at the input image, instead of performing the required reasoning steps. We present VisQA, a visual analytics tool thatexplores this question of reasoning vs. bias exploitation. It exposes the key element of state-of-the-art neural models --- attention maps in transformers. Our working hypothesis is that reasoning steps leading to model predictions are observable from attention distributions, which are particularly useful for visualization. The design process of VisQA was motivated by well-known bias examples from the fields of deep learning and vision-language reasoning and evaluated in two ways. First, as a result of a collaboration of three fields, machine learning, vision and language reasoning, and data analytics, the work lead to a better understanding of bias exploitation of neural models for VQA, which eventually resulted in an impact on its design and training through the proposition of a method for the transfer of reasoning patterns from an oracle model. Second, we also report on the design of VisQA, and a goal-oriented evaluation of VisQA targeting the analysis of a model decision process from multiple experts, providing evidence that it makes the inner workings of models accessible to users.
Complete list of metadata
Contributor : Théo Jaunet Connect in order to contact the contributor
Submitted on : Tuesday, July 20, 2021 - 5:03:28 PM
Last modification on : Tuesday, April 26, 2022 - 4:36:40 PM
Long-term archiving on: : Thursday, October 21, 2021 - 7:00:12 PM


Files produced by the author(s)



Theo Jaunet, Corentin Kervadec, Romain Vuillemot, Grigory Antipov, Moez Baccouche, et al.. VisQA: X-raying Vision and Language Reasoning in Transformers. IEEE Transactions on Visualization and Computer Graphics, Institute of Electrical and Electronics Engineers, 2021, ⟨10.1109/TVCG.2021.3114683⟩. ⟨hal-03293079⟩



Record views


Files downloads