Fine grained sport action recognition with Twin spatio-temporal convolutional neural networks

Pierre-Etienne Martin (1) , Jenny Benois-Pineau (1) , Renaud Péteri (2) , Julien Morlier (3)

1 LaBRI - Laboratoire Bordelais de Recherche en Informatique
2 MIA - Mathématiques, Image et Applications - EA 3165
3 IMS - Laboratoire de l'intégration, du matériau au système

Human action recognition in video is one of the key problems in visual data interpretation. Despite intensive research, the recognition of actions with low inter-class variability remains a challenge. This paper presents a new Siamese Spatio-Temporal Convolutional Neural Network (SSTCNN) for this purpose. When applied to table tennis, it is possible to detect and recognize 20 table tennis strokes. The model has been trained on a specific dataset, so called TTStroke-21, recorded in natural conditions at the Faculty of Sports of the University of Bordeaux. Our model takes as inputs a RGB image sequence and its computed residual Optical Flow. The proposed siamese network architecture comprises 3 spatio-temporal convolutional layers, followed by a fully connected layer where data are fused. Our method reaches an accuracy of 91.4% against 43.1% for our baseline.

Domaines: Vision par ordinateur et reconnaissance de formes [cs.CV]; Intelligence artificielle [cs.AI]; Informatique [cs]; Interface homme-machine [cs.HC]; Apprentissage [cs.LG]; Multimédia [cs.MM]; Réseau de neurones [cs.NE]; Traitement des images [eess.IV]; Traitement du signal et de l'image [eess.SP]

Pierre-Etienne Martin, Jenny Benois-Pineau, Renaud Péteri, Julien Morlier. Fine grained sport action recognition with Twin spatio-temporal convolutional neural networks. Multimedia Tools and Applications, 2020, ⟨10.1007/s11042-020-08917-3⟩. ⟨hal-02551019⟩