Fallacies of Agreement: A Critical Review of Consensus Assessment Methods for Gesture Elicitation

Theophanis Tsandilas 1, 2
1 EX-SITU - Extreme Interaction
LRI - Laboratoire de Recherche en Informatique, Inria Saclay - Ile de France
2 HCC - LRI - Human-Centered Computing (LRI)
LRI - Laboratoire de Recherche en Informatique
Abstract : Discovering gestures that gain consensus is a key goal of gesture elicitation. To this end, HCI research has developed statistical methods to reason about agreement. We review these methods and identify three major problems. First, we show that raw agreement rates disregard agreement that occurs by chance and do not reliably capture how participants distinguish among referents. Second, we explain why current recommendations on how to interpret agreement scores rely on problematic assumptions. Third, we demonstrate that significance tests for comparing agreement rates, either within or between participants, yield large Type I error rates (> 40% for α = .05). As alternatives, we present agreement indices that are routinely used in inter-rater reliability studies. We discuss how to apply them to gesture elicitation studies. We also demonstrate how to use common resampling techniques to support statistical inference with interval estimates. We apply these methods to reanalyze and reinterpret the findings of four gesture elicitation studies.
Type de document :
Article dans une revue
ACM Transactions on Computer-Human Interaction, Association for Computing Machinery, 2018, 25 (3), pp.1-49. 〈10.1145/3182168〉
Liste complète des métadonnées

Littérature citée [9 références]  Voir  Masquer  Télécharger

https://hal.archives-ouvertes.fr/hal-01788775
Contributeur : Theophanis Tsandilas <>
Soumis le : lundi 9 juillet 2018 - 22:04:41
Dernière modification le : vendredi 13 juillet 2018 - 01:11:59
Document(s) archivé(s) le : mardi 2 octobre 2018 - 06:31:15

Fichier

gelicitation-hal.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

Citation

Theophanis Tsandilas. Fallacies of Agreement: A Critical Review of Consensus Assessment Methods for Gesture Elicitation. ACM Transactions on Computer-Human Interaction, Association for Computing Machinery, 2018, 25 (3), pp.1-49. 〈10.1145/3182168〉. 〈hal-01788775v2〉

Partager

Métriques

Consultations de la notice

222

Téléchargements de fichiers

146