ModDrop: adaptive multi-modal gesture recognition

Abstract : We present a method for gesture detection and localisation based on multi-scale and multi-modal deep learning. Each visual modality captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at three temporal scales. Key to our technique is a training strategy which exploits: i) careful initialization of individual modalities; and ii) gradual fusion involving random dropping of separate channels (dubbed "ModDrop") for learning cross-modality correlations while preserving uniqueness of each modality-specific representation. We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams. Fusing multiple modalities at several spatial and temporal scales leads to a significant increase in recognition rates, allowing the model to compensate for errors of the individual classifiers as well as noise in the separate channels. Futhermore, the proposed ModDrop training technique ensures robustness of the classifier to missing signals in one or several channels to produce meaningful predictions from any number of available modalities. In addition, we demonstrate the applicability of the proposed fusion scheme to modalities of arbitrary nature by experiments on the same dataset augmented with audio.
Document type :
Journal articles
Complete list of metadatas

Cited literature [61 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01178733
Contributor : Christian Wolf <>
Submitted on : Tuesday, July 21, 2015 - 10:48:55 AM
Last modification on : Tuesday, February 26, 2019 - 4:35:39 PM
Long-term archiving on : Thursday, October 22, 2015 - 10:34:15 AM

File

main.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-01178733, version 1

Citation

Natalia Neverova, Christian Wolf, Graham W. Taylor, Florian Nebout. ModDrop: adaptive multi-modal gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, Institute of Electrical and Electronics Engineers, 2016. ⟨hal-01178733⟩

Share

Metrics

Record views

372

Files downloads

427