A. Aker, M. El-haj, D. Albakour, and U. Kruschwitz, Assessing crowdsourcing quality through objective tasks, Proceedings of LREC'12, 2012.

O. Alonso and S. Mizzaro, Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment, Proceedings of SIGIR '09: Workshop on The Future of IR Evaluation, 2009.

O. Alonso, D. E. Rose, and B. Stewart, Crowdsourcing for relevance evaluation, ACM SIGIR Forum, vol.42, issue.2, pp.9-15, 2008.
DOI : 10.1145/1480506.1480508

R. Artstein and M. Poesio, Inter-Coder Agreement for Computational Linguistics, Computational Linguistics, vol.27, issue.1, pp.555-596, 2008.
DOI : 10.1037/0033-2909.103.3.374

M. S. Bernstein, D. R. Karger, R. C. Miller, and J. Brandt, Analytic methods for optimizing realtime crowdsourcing, 2012.

V. Bhardwaj, R. Passonneau, A. Salleb-aouissi, and N. Ide, Anveshan: A tool for analysis of multiple annotators' labeling behavior, Proceedings of the 4th linguistic annotation workshop, 2010.

J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller et al., Vizwiz: nearly real-time answers to visual questions, Proceedings of the 23nd annual ACM symposium on user interface software and technology, 2010.

H. Bonneau-maynard, S. Rosset, C. Ayache, A. Kuhn, and D. Mostefa, Semantic annotation of the French media dialog corpus, Proceedings of InterSpeech, 2005.

C. Callison-burch, Fast, cheap, and creative, Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 1, EMNLP '09, 2009.
DOI : 10.3115/1699510.1699548

C. Callison-burch and M. Dredze, Creating speech and language data with Amazon's Mechanical Turk, CSLDAMT '10: Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 2010.

J. Carletta, Assessing agreement on classification tasks: the kappa statistic, Computational Linguistics, vol.22, pp.249-254, 1996.

J. Chamberlain, U. Kruschwitz, and M. Poesio, Constructing an anaphorically annotated corpus with non-experts, Proceedings of the 2009 Workshop on The People's Web Meets NLP Collaboratively Constructed Semantic Resources, People's Web '09, 2009.
DOI : 10.3115/1699765.1699774

J. Chamberlain, U. Kruschwitz, and M. Poesio, Motivations for participation in socially networked collective intelligence systems, Proceedings of CI2012, 2012.

J. Chamberlain, M. Poesio, and U. Kruschwitz, Phrase Detectives: A web-based collaborative annotation game, Proceedings of the International Conference on Semantic Systems (I- Semantics'08), 2008.

J. Chamberlain, M. Poesio, and U. Kruschwitz, A new life for a dead parrot: Incentive structures in the Phrase Detectives game, Proceedings of the WWW 2009 Workshop on Web Incentives, 2009.

T. Chklovski and Y. Gil, Improving the design of intelligent acquisition interfaces for collecting world knowledge from web contributors, Proceedings of the 3rd international conference on Knowledge capture , K-CAP '05, 2005.
DOI : 10.1145/1088622.1088630

T. Chklovski, Collecting paraphrase corpora from volunteer contributors, Proceedings of the 3rd international conference on Knowledge capture , K-CAP '05, 2005.
DOI : 10.1145/1088622.1088644
URL : http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.512.8466

J. Cohen, A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement, vol.20, issue.1, pp.37-46, 1960.
DOI : 10.1177/001316446002000104

M. Csikszentmihalyi, Flow., 1990.
DOI : 10.1037/10518-188
URL : https://hal.archives-ouvertes.fr/hal-01470857

S. Dandapat, P. Biswas, M. Choudhury, and K. Bali, Complex linguistic annotation -No easy way out! A case from Bangla and Hindi POS labeling tasks, Proceedings of the 3rd ACL Linguistic Annotation Workshop, 2009.

C. Fellbaum, WordNet: An Electronic Lexical Database, 1998.

D. Feng, S. Besana, and R. Zajac, Acquiring high quality non-expert knowledge from ondemand workforce, Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources, 2009.

F. Fenouillet, J. Kaplan, and N. Yennek, Serious games et motivation Jeux Serieux: conception et usages, 4eme Conference francophone sur les Environnements Informatiques pour l'Apprentissage Humain (EIAH'09), 2009.

K. Fort, G. Adda, and K. B. Cohen, Amazon Mechanical Turk: Gold mine or coal mine? Computational Linguistics, pp.413-420, 2011.
DOI : 10.1162/coli_a_00057
URL : http://doi.org/10.1162/coli_a_00057

K. Fort and B. Sagot, Influence of pre-annotation on POS-tagged corpus development, Proceedings of the 4th ACL Linguistic Annotation Workshop, 2010.
URL : https://hal.archives-ouvertes.fr/hal-00484294

D. Gillick and Y. Liu, Non-expert evaluation of summarization systems is risky, Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 2010.

R. Glott, P. Schmidt, and R. Ghosh, Wikipedia survey ? Overview of results, UNU-MERIT, pp.1-11, 2010.

N. Green, P. Breimyer, V. Kumar, and N. F. Samatova, Packplay: Mining semantic data in collaborative games, Proceedings of the 4th Linguistic Annotation Workshop, 2010.

B. Hladká, J. Mírovsk-`-mírovsk-`-y, and P. Schlesinger, Play the language, Proceedings of the ACL-IJCNLP 2009 Conference Short Papers on, ACL-IJCNLP '09, 2009.
DOI : 10.3115/1667583.1667648

J. Hong and C. F. Baker, How good is the crowd at " real " WSD?, Proceedings of the 5th Linguistic Annotation Workshop, 2011.

J. Howe, Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business, 2008.

P. Ipeirotis, Analyzing the Amazon Mechanical Turk marketplace CeDER Working Papers, http://hdl.handle.net, 2010.

N. L. Johnson, S. Rasmussen, C. Joslyn, L. Rocha, S. Smith et al., Symbiotic Intelligence: Self-organizing knowledge on distributed networks driven by human interaction, Proceedings of the 6th International Conference on Artificial Life, 1998.

A. Joubert and M. Lafourcade, Jeuxdemots : Un prototype ludique pour l'´ emergence de relations entre termes, Proceedings of JADT'2008, Ecole normale supérieure Lettres et sciences humaines, 2008.
URL : https://hal.archives-ouvertes.fr/lirmm-00358848

D. Jurafsky and J. H. Martin, Speech and Language Processing-2 nd edition, 2008.

B. Kanefsky, N. Barlow, and V. Gulick, Can distributed volunteers accomplish massive data analysis tasks? Lunar and Planetary Science, XXXII, 2001.

G. Kazai, In Search of Quality in Crowdsourcing for Search Engine Evaluation, Proceedings of the 33 rd European Conference on Information Retrieval, 2011.
DOI : 10.1007/978-3-642-14556-8_16

G. Kazai, N. Milic-frayling, and J. Costello, Towards methods for the collective gathering and quality control of relevance assessments, Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, SIGIR '09, 2009.
DOI : 10.1145/1571941.1572019

A. Koller, K. Striegnitz, A. Gargett, D. Byron, J. Cassell et al., Report on the 2nd NLG challenge on generating instructions in virtual environments (GIVE- 2), Proceedings of the 6th INLG, 2010.

R. Koster, A Theory of Fun for Game Design, 2005.

M. Lafourcade, Making people play for lexical acquisition, Proceedings SNLP 2007, 7th Symposium on Natural Language Processing, 2007.
URL : https://hal.archives-ouvertes.fr/lirmm-00200883

M. Lafourcade and A. Joubert, A new dynamic approach for lexical networks evaluation, Proceedings of LREC'12: 8th International Conference on Language Resources and Evaluation, 2012.
URL : https://hal.archives-ouvertes.fr/lirmm-00711483

D. Laniado, C. Castillo, A. Kaltenbrunner, and M. Fuster, Emotions and dialogue in a peer-production community, Proceedings of the Eighth Annual International Symposium on Wikis and Open Collaboration, WikiSym '12, 2012.
DOI : 10.1145/2462932.2462944

H. Lieberman, S. D. , and A. Teeters, Common consensus: A web-based game for collecting commonsense goals, Proceedings of IUI, 2007.

T. Malone, R. Laubacher, and C. Dellarocas, Harnessing Crowds: Mapping the Genome of Collective Intelligence, SSRN Electronic Journal, 2009.
DOI : 10.2139/ssrn.1381502

A. Marchetti, M. Tesconi, F. Ronzano, M. Rosella, and S. Minutol, SemKey: A semantic collaborative tagging system, Proceedings of WWW 2007 Workshop on Tagging and Metadata for Social Information Organization, 2007.

M. Marcus, B. Santorini, and M. A. Marcinkiewicz, Building a large annotated corpus of English : The Penn Treebank, Computational Linguistics, vol.19, issue.2, pp.313-330, 1993.

M. Marge, S. Banerjee, and A. I. Rudnicky, Using the Amazon Mechanical Turk for transcription of spoken language, 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, 2010.
DOI : 10.1109/ICASSP.2010.5494979

W. Mason and D. J. Watts, Financial incentives and the " performance of crowds, Proceedings of the ACM SIGKDD Workshop on Human Computation, 2009.

R. Mihalcea and T. Chklovski, Open Mind Word Expert: Creating large annotated data collections with web users help, Proceedings of the EACL 2003 Workshop on Linguistically Annotated Corpora, 2003.

J. Mrozinski, E. Whittaker, and S. Furui, Collecting a why-question corpus for development and evaluation of an automatic QA-system, Proceedings of ACL-08: HLT, 2008.

O. Nov, What motivates Wikipedians?, Communications of the ACM, vol.50, issue.11, pp.60-64, 2007.
DOI : 10.1145/1297797.1297798

S. Novotney and C. Callison-burch, Cheap, fast and good enough: Automatic speech recognition with non-expert transcription, Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2010.

M. Poesio, Discourse annotation and semantic annotation in the GNOME corpus, Proceedings of the 2004 ACL Workshop on Discourse Annotation, DiscAnnotation '04, 2004.
DOI : 10.3115/1608938.1608948

M. Poesio and R. Artstein, Anaphoric annotation in the ARRAU corpus, LREC'08, 2008.

M. Poesio, J. Chamberlain, U. Kruschwitz, L. Robaldo, and L. Ducceschi, The Phrase Detective multilingual corpus, release 0.1, Proceedings of LREC'12 Workshop on Collaborative Resource Development and Delivery, 2012.

M. Poesio, J. Chamberlain, U. Kruschwitz, L. Robaldo, and L. Ducceschi, Phrase detectives, ACM Transactions on Interactive Intelligent Systems, vol.3, issue.1, 2012.
DOI : 10.1145/2448116.2448119

M. Poesio, P. Sturt, R. Arstein, and R. Filik, Underspecification and Anaphora: Theoretical Issues and Preliminary Evidence, Discourse Processes, pp.157-175, 2006.
DOI : 10.1016/S0749-596X(02)00536-3

M. Poesio and R. Vieira, A corpus-based investigation of definite description use, Computational Linguist, vol.24, issue.2, 1998.

A. Quinn and B. Bederson, Human computation, Proceedings of the 2011 annual conference on Human factors in computing systems, CHI '11, 2011.
DOI : 10.1145/1978942.1979148

W. Rafelsberger and A. Scharl, Games with a purpose for social networking platforms, Proceedings of the 20th ACM conference on Hypertext and hypermedia, HT '09, 2009.
DOI : 10.1145/1557914.1557948

J. Ross, L. Irani, M. S. Silberman, A. Zaldivar, and B. Tomlinson, Who are the crowdworkers?, Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems, CHI EA '10, 2010.
DOI : 10.1145/1753846.1753873

K. Siorpaes and M. Hepp, Games with a Purpose for the Semantic Web, IEEE Intelligent Systems, vol.23, issue.3, pp.50-60, 2008.
DOI : 10.1109/MIS.2008.45

F. Smadja, Mixing financial, social and fun incentives for social voting. World Wide Web Internet And Web Information Systems, 2009.

R. Snow, B. O-'connor, D. Jurafsky, and A. Y. Ng, Cheap and fast---but is it good?, Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP '08, 2008.
DOI : 10.3115/1613715.1613751

J. Surowiecki, The Wisdom of Crowds, Anchor, 2005.

P. Sweetser and P. Wyeth, GameFlow, Computers in Entertainment, vol.3, issue.3, 2005.
DOI : 10.1145/1077246.1077253

S. Thaler, K. Siorpaes, E. Simperl, and C. Hofer, A survey on games for knowledge acquisition, 2011.

S. Tratz and E. Hovy, A taxonomy, dataset, and classifier for automatic noun compound interpretation, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 2010.

L. Von-ahn, Games with a Purpose, Computer, vol.39, issue.6, pp.92-94, 2006.
DOI : 10.1109/MC.2006.196

L. Von-ahn and L. Dabbish, Labeling images with a computer game, Proceedings of the 2004 conference on Human factors in computing systems , CHI '04, 2004.
DOI : 10.1145/985692.985733

L. Von-ahn and L. Dabbish, Designing games with a purpose, Communications of the ACM, vol.51, issue.8, pp.58-67, 2008.
DOI : 10.1145/1378704.1378719

L. Von-ahn, R. Liu, and M. Blum, Peekaboom, Proceedings of the SIGCHI conference on Human Factors in computing systems , CHI '06, 2006.
DOI : 10.1145/1124772.1124782

P. Wais, S. Lingamneni, D. Cook, J. Fennell, B. Goldenberg et al., Towards building a high-quality workforce with Mechanical Turk, Proceedings of Computational Social Science and the Wisdom of Crowds (NIPS), 2010.

A. Wang, C. D. Hoang, and M. Y. Kan, Perspectives on crowdsourcing annotations for natural language processing, Language Resources and Evaluation, vol.10, issue.6, pp.1-19, 2010.
DOI : 10.1007/s10579-012-9176-1

A. W. Woolley, C. F. Chabris, A. Pentland, N. Hashmi, and T. W. Malone, Evidence for a Collective Intelligence Factor in the Performance of Human Groups, Science, vol.330, issue.6004, pp.686-688, 2010.
DOI : 10.1126/science.1193147

H. Yang and C. Lai, Motivations of Wikipedia content contributors, Computers in Human Behavior, vol.26, issue.6, 2010.
DOI : 10.1016/j.chb.2010.04.011

M. Yuen, L. Chen, and I. King, A survey of human computation systems playing / having fun, Information Sciences, 2009.