Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings: Application to Personality Traits

Baiyu Chen 1 Sergio Escalera 2 Isabelle Guyon 3, 4 Victor Ponce-Lopez 2 Nihar Shah 5 Marc Oliu 2
3 TAO - Machine Learning and Optimisation
CNRS - Centre National de la Recherche Scientifique : UMR8623, Inria Saclay - Ile de France, UP11 - Université Paris-Sud - Paris 11, LRI - Laboratoire de Recherche en Informatique
Abstract : We address the problem of calibration of workers whose task is to label patterns with continuous variables, which arises for instance in labeling images of videos of humans with continuous traits. Worker bias is particularly difficult to evaluate and correct when many workers contribute just a few labels, a situation arising typically when labeling is crowd-sourced. In the scenario of labeling short videos of people facing a camera with personality traits, we evaluate the feasibility of the pairwise ranking method to alleviate bias problems. Workers are exposed to pairs of videos at a time and must order by preference. The variable levels are reconstructed by fitting a Bradley-Terry-Luce model with maximum likelihood. This method may at first sight, seem prohibitively expensive because for N videos, p=N(N−1)/2 pairs must be potentially processed by workers rather that N videos. However, by performing extensive simulations, we determine an empirical law for the scaling of the number of pairs needed as a function of the number of videos in order to achieve a given accuracy of score reconstruction and show that the pairwise method is affordable. We apply the method to the labeling of a large scale dataset of 10,000 videos used in the ChaLearn Apparent Personality Trait challenge.
Liste complète des métadonnées

Contributor : Isabelle Guyon <>
Submitted on : Friday, October 14, 2016 - 9:29:56 AM
Last modification on : Thursday, April 5, 2018 - 12:30:12 PM



Baiyu Chen, Sergio Escalera, Isabelle Guyon, Victor Ponce-Lopez, Nihar Shah, et al.. Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings: Application to Personality Traits. European Conference on Computer Vision (ECCV 2016) Workshops, Oct 2016, Amsterdam, Netherlands. pp.419-432, ⟨10.1007/978-3-319-49409-8_33⟩. ⟨hal-01381148⟩



Record views