THORN: Temporal Human-Object Relation Network for Action Recognition

Mohammed Guermal (1) , Rui Dai (2, 1) , Francois F Bremond (1, 2)

1 STARS - Spatio-Temporal Activity Recognition Systems
2 UniCA - Université Côte d'Azur

Most action recognition models treat human activities as unitary events. However, human activities often follow a certain hierarchy. In fact, many human activities are compositional. Also, these actions are mostly human-object interactions. In this paper we propose to recognize human action by leveraging the set of interactions that define an action. In this work, we present an end-to-end network: THORN, that can leverage important human-object and object-object interactions to predict actions. This model is built on top of a 3D backbone network. The key components of our model are: 1) An object representation filter for modeling object. 2) An object relation reasoning module to capture object relations. 3) A classification layer to predict the action labels. To show the robustness of THORN, we evaluate it on EPIC-Kitchen55 and EGTEA Gaze+, two of the largest and most challenging first-person and human-object interaction datasets. THORN achieves state-of-the-art performance on both datasets.

Mots clés: Activity recognition and understanding; Human object interaction recognition; Video analytics

Domaines: Informatique [cs]; Vision par ordinateur et reconnaissance de formes [cs.CV]

Mohammed Guermal, Rui Dai, Francois F Bremond. THORN: Temporal Human-Object Relation Network for Action Recognition. ICPR 2022 - International Conference on Pattern Recognition, Aug 2022, Montreal, Canada. ⟨hal-03698623⟩