Learning from narrated instruction videos

Jean-Baptiste Alayrac (1, 2, 3) , Piotr Bojanowski (1, 2, 3) , Nishant Agrawal (1, 2, 3) , Josef Sivic (1, 2, 3) , Ivan Laptev (1, 2, 3) , Simon Lacoste-Julien (1, 3, 4)

1 DI-ENS - Département d'informatique - ENS Paris
2 WILLOW - Models of visual object recognition and scene understanding
3 PSL - Université Paris Sciences et Lettres
4 SIERRA - Statistical Machine Learning and Parsimony

Automatic assistants could guide a person or a robot in performing new tasks, such as changing a car tire or repotting a plant. Creating such assistants, however, is non-trivial and requires understanding of visual and verbal content of a video. Towards this goal, we here address the problem of automatically learning the main steps of a task from a set of narrated instruction videos. We develop a new unsupervised learning approach that takes advantage of the complementary nature of the input video and the associated narration. The method sequentially clusters textual and visual representations of a task, where the two clustering problems are linked by joint constraints to obtain a single coherent sequence of steps in both modalities. To evaluate our method, we collect and annotate a new challenging dataset of real-world instruction videos from the Internet. The dataset contains videos for five different tasks with complex interactions between people and objects, captured in a variety of indoor and outdoor settings. We experimentally demonstrate that the proposed method can automatically discover, learn and localize the main steps of a task in input videos.

Mots clés: Unsupervised learning; Narrated instruction videos; Step discovery

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Intelligence artificielle [cs.AI]

Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, et al.. Learning from narrated instruction videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, XX. ⟨hal-01580630⟩