Automatic Segmentation of Sign Language into Subtitle-Units

Hannah Bull (1) , Michèle Gouiffès (2) , Annelies Braffort (3)

1 LISN - Laboratoire Interdisciplinaire des Sciences du Numérique
2 AMI - Architectures et Modèles pour l'Interaction - LISN
3 ILES - Information, Langue Ecrite et Signée

We present baseline results for a new task of automatic segmentation of Sign Language video into sentence-like units. We use a corpus of natural Sign Language video with accurately aligned subtitles to train a spatio-temporal graph convolutional network with a BiLSTM on 2D skeleton data to automatically detect the temporal boundaries of subtitles. In doing so, we segment Sign Language video into subtitle-units that can be translated into phrases in a written language. We achieve a ROC-AUC statistic of 0.87 at the frame level and 92% label accuracy within a time margin of 0.6s of the true labels.

Mots clés: Sign Language; Segmentation; Sentence; Subtitle; Graph Neural Network; Skeleton Keypoints

Domaines: Intelligence artificielle [cs.AI]; Vision par ordinateur et reconnaissance de formes [cs.CV]

Hannah Bull, Michèle Gouiffès, Annelies Braffort. Automatic Segmentation of Sign Language into Subtitle-Units. Sign Language Recognition, Translation & Production workshop, Aug 2020, Glasgow ( virtual ), United Kingdom. ⟨10.1007/978-3-030-66096-3_14⟩. ⟨hal-03098684⟩