L-STAP: Learned Spatio-Temporal Adaptive Pooling for Video Captioning - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2019

L-STAP: Learned Spatio-Temporal Adaptive Pooling for Video Captioning

Danny Francis
  • Fonction : Auteur
  • PersonId : 1125186

Résumé

Automatic video captioning can be used to enrich TV programs with textual informations on scenes. These informations can be useful for visually impaired people, but can also be used to enhance indexing and research of TV records. Video captioning can be seen as being more challenging than image captioning. In both cases, we have to tackle a challenging task where a visual object has to be analyzed, and translated into a textual description in natural language. However, analyzing videos requires not only to parse still images, but also to draw correspondences through time. Recent works in video captioning have intended to deal with these issues by separating spatial and temporal analysis of videos. In this paper, we propose a Learned Spatio-Temporal Adaptive Pooling (L-STAP) method that combines spatial and temporal analysis. More specifically, we first process a video frame-by-frame through a Convolutional Neural Network. Then, instead of applying an average pooling operation to reduce dimensionality, we apply our L-STAP, which attends to specific regions in a given frame based on what appeared in previous frames. Experiments on MSVD and MSR-VTT datasets show that our method outperforms state-of-the-art methods on the video captioning task in terms of several evaluation metrics.
Fichier principal
Vignette du fichier
publi-6051 (1).pdf (1.49 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03555313 , version 1 (03-02-2022)

Identifiants

Citer

Danny Francis, Benoit Huet. L-STAP: Learned Spatio-Temporal Adaptive Pooling for Video Captioning. AI4TV 2019, 1st International Workshop on AI for Smart TV Content Production, Access and Delivery, Oct 2019, Nice, France. pp.33-41, ⟨10.1145/3347449.3357484⟩. ⟨hal-03555313⟩

Collections

EURECOM ANR
26 Consultations
24 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More