Skip to Main content Skip to Navigation
Conference papers

Filtering with the Crowd: CrowdScreen revisited (technical report)

Abstract : Filtering a set of items, based on a set of properties that can be verified by humans, is a common application of CrowdSourcing. When the workers are error-prone, each item is presented to multiple users, to limit the probability of misclassification. Since the Crowd is a relatively expensive resource, minimizing the number of questions per item may naturally result in big savings. Several algorithms to address this minimization problem have been presented in the CrowdScreen framework by Parameswaran et al. However, those algorithms do not scale well and therefore cannot be used in scenarios where high accuracy is required in spite of high user error rates. The goal of this paper is thus to devise algorithms that can cope with such situations. To achieve this, we provide new theoretical insights to the problem, then use them to develop a new efficient algorithm. We also propose novel optimizations for the algorithms of CrowdScreen that improve their scalability. We complement our theoretical study by an experimental evaluation of the algorithms on a large set of synthetic parameters as well as real-life crowdsourcing scenarios, demonstrating the advantages of our solution.
Complete list of metadatas

Cited literature [20 references]  Display  Hide  Download

https://hal.archives-ouvertes.fr/hal-01239458
Contributor : Benoit Groz <>
Submitted on : Monday, December 7, 2015 - 6:02:05 PM
Last modification on : Wednesday, September 16, 2020 - 5:45:31 PM
Long-term archiving on: : Saturday, April 29, 2017 - 8:42:31 AM

File

glmm-icdt16-long.pdf
Files produced by the author(s)

Licence


Distributed under a Creative Commons Attribution 4.0 International License

Identifiers

Citation

Benoît Groz, Ezra Levin, Isaac Meilijson, Tova Milo. Filtering with the Crowd: CrowdScreen revisited (technical report). 19th International Conference on Database Theory (ICDT 2016), Mar 2016, Bordeaux, France. pp.12:1--12:18, ⟨10.4230/LIPIcs.ICDT.2016.12⟩. ⟨hal-01239458⟩

Share

Metrics

Record views

687

Files downloads

228