HAL will be down for maintenance from Friday, June 10 at 4pm through Monday, June 13 at 9am. More information
Skip to Main content Skip to Navigation
Conference papers

Weakly-Supervised Alignment of Video With Text

Piotr Bojanowski 1, 2 Rémi Lajugie 2, 3 Edouard Grave 4 Francis Bach 2, 3 Ivan Laptev 1, 2 Jean Ponce 1, 2 Cordelia Schmid 5
1 WILLOW - Models of visual object recognition and scene understanding
CNRS - Centre National de la Recherche Scientifique : UMR8548, Inria Paris-Rocquencourt, DI-ENS - Département d'informatique - ENS Paris
3 SIERRA - Statistical Machine Learning and Parsimony
DI-ENS - Département d'informatique - ENS Paris, Inria Paris-Rocquencourt, CNRS - Centre National de la Recherche Scientifique : UMR8548
5 LEAR - Learning and recognition in vision
Inria Grenoble - Rhône-Alpes, LJK - Laboratoire Jean Kuntzmann, Grenoble INP - Institut polytechnique de Grenoble - Grenoble Institute of Technology
Abstract : Suppose that we are given a set of videos, along with natural language descriptions in the form of multiple sentences (e.g., manual annotations, movie scripts, sport summaries etc.), and that these sentences appear in the same temporal order as their visual counterparts. We propose in this paper a method for aligning the two modalities, i.e., automatically providing a time stamp for every sentence. Given vectorial features for both video and text, we propose to cast this task as a temporal assignment problem, with an implicit linear mapping between the two feature modalities. We formulate this problem as an integer quadratic program, and solve its continuous convex relaxation using an efficient conditional gradient algorithm. Several rounding procedures are proposed to construct the final integer solution. After demonstrating significant improvements over the state of the art on the related task of aligning video with symbolic labels [7], we evaluate our method on a challenging dataset of videos with associated textual descriptions [36], using both bag-of-words and continuous representations for text.
Document type :
Conference papers
Complete list of metadata

Contributor : Piotr Bojanowski Connect in order to contact the contributor
Submitted on : Friday, May 22, 2015 - 11:36:58 AM
Last modification on : Thursday, March 17, 2022 - 10:08:44 AM
Long-term archiving on: : Thursday, April 20, 2017 - 6:40:54 AM


Files produced by the author(s)


  • HAL Id : hal-01154523, version 1
  • ARXIV : 1505.06027


Piotr Bojanowski, Rémi Lajugie, Edouard Grave, Francis Bach, Ivan Laptev, et al.. Weakly-Supervised Alignment of Video With Text. ICCV 2015 - IEEE International Conference on Computer Vision, Dec 2015, Santiago, Chile. ⟨hal-01154523v1⟩



Record views


Files downloads