Measuring the Expertise of Workers for Crowdsourcing Applications

Abstract : Crowdsourcing platforms enable companies to propose tasks to a large crowd of users. The workers receive a compensation for their work according to the serious of the tasks they managed to accomplish. The evaluation of the quality of responses obtained from the crowd remains one of the most important problems in this context. Several methods have been proposed to estimate the expertise level of crowd workers. We propose an innovative measure of expertise assuming that we possess a dataset with an objective comparison of the items concerned. Our method is based on the definition of four factors with the theory of belief functions. We compare our method to the Fagin distance on a dataset from a real experiment, where users have to assess the quality of some audio recordings. Then, we propose to fuse both the Fagin distance and our expertise measure.
Document type :
Book sections
Complete list of metadatas

Cited literature [14 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-02162937
Contributor : Arnaud Martin <>
Submitted on : Sunday, June 23, 2019 - 8:57:07 AM
Last modification on : Wednesday, June 26, 2019 - 1:36:38 AM

Files

AKDM_final.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-02162937, version 1
  • ARXIV : 1907.10588

Citation

Jean-Christophe Dubois, Laetitia Gros, Mouloud Kharoune, Yolande Le Gall, Arnaud Martin, et al.. Measuring the Expertise of Workers for Crowdsourcing Applications. Advances in Knowledge Discovery and Management, pp.139-157, 2019. ⟨hal-02162937⟩

Share

Metrics

Record views

46

Files downloads

31