Multimodal stress detection from multiple assessments

Jonathan Aigrain (1) , Michel Spodenkiewicz (2, 3, 4, 1) , Severine Dubuisson (1, 5, 6) , Marcin Detyniecki (7, 8) , David Cohen (1, 2) , Mohamed Chetouani (1)

1 ISIR - Institut des Systèmes Intelligents et de Robotique
2 CHU Pitié-Salpêtrière [AP-HP]
3 CHU Sud Saint Pierre [Ile de la Réunion]
4 SMSP - U1178 - Santé mentale et santé publique
5 LIS - Laboratoire d'Informatique et des Systèmes (LIS) (Marseille, Toulon)
6 I&M - Images et Modèles
7 LFI - Learning, Fuzzy and Intelligent systems
8 PAN - Polska Akademia Nauk = Polish Academy of Sciences = Académie polonaise des sciences

Stress is a complex phenomenon that impacts the body and the mind at several levels. It has been studied for more than a century from different perspectives, which result in different definitions and different ways to assess the presence of stress. This paper introduces a methodology for analyzing multimodal stress detection results by taking into account the variety of stress assessments. As a first step, we have collected video, depth and physiological data from 25 subjects in a stressful situation: a socially evaluated mental arithmetic test. As a second step, we have acquired 3 different assessments of stress: self-assessment, assessments from external observers and assessment from a physiology expert. Finally, we extract 101 behavioural and physiological features and evaluate their predictive power for the 3 collected assessments using a classification task. Using multimodal features, we obtain average F1 scores up to 0.85. By investigating the composition of the best selected feature subsets and the individual feature classification performances, we show that several features provide valuable information for the classification of the 3 assessments: features related to body movement, blood volume pulse and heart rate. From a methodological point of view, we argue that a multiple assessment approach provide more robust results.

Mots clés: Index Terms—Stress; assessment; behaviour; physiology; multimodal; classification

Domaines: Informatique [cs]

Jonathan Aigrain, Michel Spodenkiewicz, Severine Dubuisson, Marcin Detyniecki, David Cohen, et al.. Multimodal stress detection from multiple assessments. IEEE Transactions on Affective Computing, 2016, PP (99), pp.1 - 1. ⟨10.1109/TAFFC.2016.2631594⟩. ⟨hal-01416517⟩