, org/) v3.5.2. The following packages are used: 1

, A.2 REFERENCES CREX and CrowdED can be found on Figshare associated to the following DOI

,

A. , CONFIGURATIONS Following is a list of the configurable parameters of CREX along with their descriptions: A

M. Alsayasneh, Personalized and Diverse Task Composition in Crowdsourcing, IEEE Transactions on Knowledge and Data Engineering, vol.30
DOI : 10.1109/tkde.2017.2755660
URL : https://hal.archives-ouvertes.fr/hal-02000487

, IEEE, vol.80, p.103, 2018.

V. Ambati, S. Vogel, and J. Carbonell, Towards Task Recommendation in Micro-Task Markets, AAAI Conference on Human Computation, vol.30, 2011.

S. Amer-yahia, Task composition in crowdsourcing, IEEE International Conference on Data Science and Advanced Analytics (DSAA). IEEE. 2016, pp.194-203
URL : https://hal.archives-ouvertes.fr/hal-01407780

. Apisense and . Url,

L. Azzopardi, M. Girolami, and K. Van-risjbergen, Investigating the relationship between language model perplexity and IR precision-recall measures, International ACM SIGIR conference on research and development in information retrieval, pp.369-370, 2003.

Y. Baba and H. Kashima, Statistical quality estimation for general crowdsourcing tasks, Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp.554-562, 2013.

M. Balabanovi´cbalabanovi´c and Y. Shoham, Fab: content-based, collaborative recommendation, Communications of the ACM, vol.40, issue.3, pp.66-72, 1997.

P. Bansal, C. Eickhoff, and T. Hofmann, Active content-based crowdsourcing task selection, Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp.529-538, 2016.

J. A. Bargh and . Chartrand, Studying the mind in the middle: A practical guide to priming and automaticity research. Handbook of research methods in social psychology, 2000.

. Michael-s-bernstein, Crowd-powered systems, KI-Künstliche Intelligenz, pp.69-73, 2013.

M. David, . Blei, Y. Andrew, and M. Ng, Latent dirichlet allocation, Journal of Machine Learning Research, vol.3, pp.993-1022, 2003.

S. Cayzer and U. Aickelin, Recommender system based on the immune network, p.19, 2002.

X. Chen, Q. Lin, and D. Zhou, Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing, International Conference on Machine Learning, pp.64-72, 2013.

A. P. Dawid and A. M. Skene, Maximum Likelihood Estimation of Observer ErrorRates Using the EM Algorithm, English. In: Journal of the Royal Statistical Society. Series C (Applied Statistics), vol.28, pp.20-28, 1979.

P. Arthur, N. M. Dempster, D. Laird, and . Rubin, Maximum likelihood from incomplete data via the EM algorithm, Journal of the royal statistical society. Series B (methodological, pp.1-38, 1977.

J. Deng, Imagenet: A large-scale hierarchical image database, IEEE Conference on Computer Vision and Pattern Recognition, pp.248-255, 2009.

B. Dhingra, Tweet2vec: Character-based distributed representations for social media, p.93, 2016.

G. Djellel-eddine-difallah, P. Demartini, and . Cudré-mauroux, Picka-crowd: tell me what you like, and i'll tell you what to do, Proceedings of the 22nd international conference on World Wide Web. ACM. 2013, vol.32, pp.367-374

. Djellel-eddine-difallah, Scaling-up the crowd: Micro-task pricing schemes for worker retention and latency improvement, Second AAAI Conference on Human Computation and Crowdsourcing, vol.24, p.20, 2014.

. Djellel-eddine-difallah, The dynamics of micro-task crowdsourcing: The case of amazon mturk, Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, pp.238-247, 2015.

A. Doan, R. Ramakrishnan, and A. Halevy, Crowdsourcing systems on the world-wide web, Communications of the ACM, vol.54, issue.4, pp.86-96, 2011.

, Amazon Mechanical Turk documentation, vol.32, p.16

, Guide To: Contributors-Geography/Language Page, pp.5-8

, Guide to: Contributors Page, pp.5-8

, Test Question Best Practices, pp.5-8

, Test Question Enabling and Quiz Mode, pp.5-8

S. Dow, Shepherding the crowd yields better work, Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, pp.1013-1022, 2012.

T. Susan and . Dumais, Latent semantic analysis". In: Annual review of information science and technology, vol.38, pp.188-230, 2004.

. Facebook and . Url,

R. Feldman and J. Sanger, The text mining handbook: advanced approaches in analyzing unstructured data, p.103, 2007.

. Figureeight and . Url,

A. Finnerty, Keep it simple: Reward and task design in crowdsourcing, Proceedings of the Biannual Conference of the Italian Chapter, p.14, 2013.

E. A. Fleishman, Toward a taxonomy of human performance, American Psychologist, pp.1127-1149, 1957.

. Freelancer and . Url, , vol.30

. Inc and . Gartner, Emerging Technology Hype Cycle for 2017, pp.20-23, 2017.

D. Geiger and M. Schader, Personalized task recommendation in crowdsourcing information systems Current state of the art". In: Decision Support Systems, vol.65, pp.3-16, 2014.

. Gensim, , pp.2018-2020

A. Ghosh, S. Kale, and P. Mcafee, Who moderates the moderators?: crowdsourcing abuse detection in user-generated content, Proceedings of the 12th ACM conference on Electronic commerce, pp.167-176, 2011.

Y. Gil, A Controlled Crowdsourcing Approach for Practical Ontology Extensions and Metadata Annotations, International Semantic Web Conference, pp.231-246, 2017.

D. Haas, Argonaut: Macrotask Crowdsourcing for Complex Data Processing, vol.8, pp.1642-1653, 2015.

C. Harris, Youre hired! an examination of crowdsourcing incentive models in human resource tasks, Proceedings of the Workshop on Crowdsourcing for Search and Data Mining (CSDM) at the Fourth ACM International Conference on Web Search and Data Mining (WSDM), pp.15-18, 2011.

U. Hassan and E. Curry, A capability requirements approach for predicting worker performance in crowdsourcing, IEEE CollaboratCom. 2013, p.103

C. Ho, S. Jabbari, and J. W. Vaughan, Adaptive task assignment for crowdsourced classification, International Conference on Machine Learning, pp.534-542, 2013.

T. Hofmann, Probabilistic latent semantic analysis, Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence, pp.289-296, 1999.

J. Howe, Crowdsourcing: A Definition, Wired Blog Network: Crowdsourcing, 2006.

J. Howe, Crowdsourcing: How the power of the crowd is driving the future of business. Random House, 2008.

V. Nguyen-quoc and . Hung, An evaluation of aggregation techniques in crowdsourcing, International Conference on Web Information Systems Engineering, pp.1-15, 2013.

. Ibisworld, Crowdsourcing service providers, pp.20-23, 2018.

. Innocentive and . Url,

G. Panagiotis and . Ipeirotis, Demographics of mechanical turk, vol.70, p.52, 2010.

Y. Jin, Leveraging Side Information to Improve Label Quality Control in Crowd-sourcing, Human Computation. 2017

N. L. Johnson, S. I. Kotz, and N. Balakrishnan, Beta distributions, Continuous univariate distributions, pp.221-235, 1994.

A. Jøsang, R. Ismail, and C. Boyd, A survey of trust and reputation systems for online service provision". In: Decision support systems, vol.43, p.17, 2007.

H. , J. , and M. Lease, Improving Consensus Accuracy via Z-Score and Weighted Voting, Human Computation. 2011 (cited on pages 6, vol.12, p.59

E. Kanoulas, Overview of the trec 2011 session track, 2011.

D. R. Karger, S. Oh, and D. Shah, Iterative Learning for Reliable Crowdsourcing Systems, Conference on Neural Information Processing Systems, vol.12, 2011.

S. David-r-karger, D. Oh, and . Shah, Efficient crowdsourcing for multi-class labeling, ACM SIGMETRICS Performance Evaluation Review, pp.81-92, 2013.

G. Kazai, In search of quality in crowdsourcing for search engine evaluation, European Conference on Information Retrieval, pp.165-176, 2011.

G. Kazai, J. Kamps, and N. Milic-frayling, The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy, ACM International Conference on Information and Knowledge Management, pp.2583-2586, 2012.

A. Faiza-khan-khattak and . Salleb-aouissi, Quality control of crowd labeling through expert evaluation, Proceedings of the NIPS 2nd Workshop on Computational Social Science and the Wisdom of Crowds, vol.2, p.12, 2011.

A. Faiza-khan-khattak and . Salleb-aouissi, Robust crowd labeling using little expertise, International Conference on Discovery Science, pp.94-109, 2013.

. Kickstarter and . Url,

A. Kobren, Getting more for less: Optimized crowdsourcing with dynamic tasks and goals, Proceedings of the 24th international conference on world wide web. International World Wide Web Conferences Steering Committee, pp.592-602, 2015.
DOI : 10.1145/2736277.2741681
URL : http://dl.acm.org/ft_gateway.cfm?id=2741681&type=pdf

K. Krippendorff, Content Analysis. An Introduction to its Methodology, SAGE, p.105, 2012.

. Kumuluz and . Url,

J. Le, Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution, 2010 workshop on crowdsourcing for search evaluation, vol.57, pp.21-26, 2010.

J. Le, Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution, SIGIR 2010 workshop on crowdsourcing for search evaluation, vol.2126, p.22, 2010.

Q. Le and T. Mikolov, Distributed representations of sentences and documents, International Conference on Machine Learning, vol.75, p.43, 2014.

M. Lease, On Quality Control and Machine Learning in Crowdsourcing, Human Computation, vol.11, p.11, 2011.

K. Lee, J. Caverlee, and S. Webb, The social honeypot project: protecting online communities from spammers, Proceedings of the 19th international conference on World wide web. ACM. 2010, pp.1139-1140
DOI : 10.1145/1772690.1772843

S. Lee, S. Park, and S. Park, A quality enhancement of crowdsourcing based on quality evaluation and user-level task assignment framework, Big Data and Smart Computing (BIGCOMP), 2014 International Conference on. IEEE, pp.60-65, 2014.

H. Li and B. Yu, Error rate bounds and iterative weighted majority voting for crowdsourcing, vol.86, 2014.

H. Li, B. Yu, and D. Zhou, Error rate analysis of labeling by crowdsourcing, Machine Learning meets Crowdsourcing Workshop, vol.55, 2013.

H. Li, B. Zhao, and A. Fuxman, The Wisdom of Minority: Discovering and Targeting the Right Group of Workers for Crowdsourcing

K. Seoul, , vol.2, pp.978-979, 2014.

A. Mao, Volunteering versus work for pay: Incentives and tradeoffs in crowdsourcing, Human Computation. 2013 (cited on pages 20, vol.59

. Marketsandmarkets, Application (Smart Cities, Location Services, Analytics, Augmented Reality), Organization Size (SME, Large Enterprises), Vertical, and Region-Global Forecast to 2022, Computing Market by Component (Hardware, Platform, Solutions), pp.20-23, 2017.

J. Afra, L. Mashhadi, and . Capra, Quality Control for Real-time Ubiquitous Crowdsourcing, UbiCrowd. Beijing, China, pp.978-979, 2011.

P. Mavridis, D. Gross-amblard, and Z. Miklós, Using hierarchical skills for optimized task assignment in knowledge-intensive crowdsourcing, Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, vol.17, p.32, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01306481

T. Mitra, J. Clayton, E. Hutto, and . Gilbert, Comparing person-and processcentric strategies for obtaining quality data on amazon mechanical turk, Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, p.17, 2015.
DOI : 10.1145/2702123.2702553

K. Mo, E. Zhong, and Q. Yang, Cross-task Crowdsourcing". In: SIGKDD, vol.84, pp.978-979, 2013.
DOI : 10.1145/2487575.2487593
URL : http://www.cse.ust.hk/~qyang/Docs/2013/kdd2013-ctc 5June-1.pdf

. Todd-k-moon, The expectation-maximization algorithm". In: IEEE Signal processing magazine, vol.13, pp.47-60, 1996.

R. Morris, M. Dontcheva, and E. Gerber, Priming for Better Performance in Microtask Crowdsourcing Environments, IEEE Internet Computing, vol.16, issue.5, pp.13-19, 2012.

H. Mousa, PrivaSense: Privacy-Preserving and Reputation-Aware Mobile Participatory Sensing, Proceedings of the 14th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous, 2017.
DOI : 10.4108/eai.7-11-2017.2273602
URL : https://hal.archives-ouvertes.fr/hal-01952294

G. Derek and . Murray, The case for crowd computing, Proceedings of the second ACM SIGCOMM workshop on Networking, systems, and applications on mobile handhelds. ACM. 2010, pp.39-44

N. Naveed, Searching microblogs: coping with sparsity and document quality, Proceedings of the 20th ACM international conference on Information and knowledge management, pp.183-188, 2011.
DOI : 10.1145/2063576.2063607

J. Neyman, On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection, Journal of the Royal Statistical Society, vol.97, pp.558-625, 1934.

Q. Viet and H. Nguyen, Batc: a benchmark for aggregation techniques in crowdsourcing, International ACM SIGIR conference on research and development in information retrieval, pp.1079-1080, 2013.

D. Oleson, Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing, In: Human computation, vol.11, p.59, 2011.

. Lawrence-a-palinkas, Purposeful sampling for qualitative data collection and analysis in mixed method implementation research, vol.42, pp.533-544, 2015.

K. Parshotam, Crowd computing: a literature review and definition". In: Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference, pp.121-130, 2013.

J. Alexander, . Quinn, and . Benjamin-b-bederson, Human computation: a survey and taxonomy of a growing field, Proceedings of the SIGCHI conference on human factors in computing systems, pp.1403-1412, 2011.

. Quora and . Url,

H. Rahman, Task assignment optimization in collaborative crowdsourcing, Data Mining (ICDM), 2015 IEEE International Conference on. IEEE. 2015, vol.17, p.103
DOI : 10.1109/icdm.2015.119
URL : https://hal.archives-ouvertes.fr/hal-02000609

L. Andreevich-rastrigin, Random Search in Optimization Problems for Multiparameter Systems, p.66

C. Vikas, S. Raykar, and . Yu, Ranking annotators for crowdsourced labeling tasks, Advances in Neural Information Processing Systems 24

. Shawe-taylor, , vol.14, p.13, 2011.

C. Vikas, S. Raykar, and . Yu, Eliminating spammers and ranking annotators for crowdsourced labeling tasks, Journal of Machine Learning Research, vol.13, pp.491-518, 2012.

C. Vikas and . Raykar, Supervised Learning from Multiple Experts: Whom to Trust when Everyone Lies a Bit, vol.98, pp.12-14, 2009.

. Vikas-c-raykar, Supervised learning from multiple experts: whom to trust when everyone lies a bit, Proceedings of the 26th Annual international conference on machine learning, pp.889-896, 2009.

C. Vikas and . Raykar, Learning From Crowds", In: J. Mach. Learn. Res, vol.86, pp.1297-1322, 2010.

. Reddit and . Url,

A. Rosenberg and J. Hirschberg, V-measure: A conditional entropy-based external cluster evaluation measure, Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL), vol.77, p.62, 2007.

J. Ross, Who are the crowdworkers?: shifting demographics in mechanical turk, CHI'10 extended abstracts on Human factors in computing systems. ACM. 2010, pp.2863-2872

J. Peter, L. Rousseeuw, and . Kaufman, Finding groups in data, vol.77, p.62, 1990.

. Senjuti-basu-roy, Task assignment optimization in knowledge-intensive crowdsourcing, The VLDB booktitle, vol.24, pp.467-491, 2015.

M. Jeffrey, A. Rzeszotarski, and . Kittur, Instrumenting the Crowd: Using Implicit Behavioral Measures to Predict Task Performance, UIST, pp.13-22, 2011.

M. Safran and D. Che, Real-time recommendation algorithms for crowdsourcing systems, Applied Computing and Informatics, 2016.
DOI : 10.1016/j.aci.2016.01.001
URL : https://doi.org/10.1016/j.aci.2016.01.001

G. Salton and M. Mcgill, Modern information retrieval, p.75, 1983.

C. Sarasua, Crowdsourcing and the Semantic Web: A research manifesto, Human Computation, vol.2, pp.3-17, 2015.

J. Schafer, Collaborative filtering recommender systems, The adaptive web, pp.291-324, 2007.

D. Schall, B. Satzger, and H. Psaier, Crowdsourcing tasks to social networks in BPEL4People, World Wide Web, vol.17, pp.1-32, 2014.
DOI : 10.1007/s11280-012-0180-6

D. Schneider, CSCWD: Five characters in search of crowds, Computer Supported Cooperative Work in Design (CSCWD), 2012 IEEE 16th International Conference on. IEEE. 2012, pp.634-641

. Scikit, , pp.2018-2020

S. Singh, Blockchain Market worth 7,683.7 Million USD by 2022

R. Snow, Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks, Conference on empirical methods in natural language processing, vol.57, pp.254-263, 2008.

. Raymond-t-sparrowe, Social networks and the performance of individuals and groups, Academy of management booktitle, vol.44, p.17, 2001.

. Thingiverse and . Url,

. Thingiverse, Thingiverse-Digital Designs for Physical Object-About, pp.5-8

T. Tian and J. Zhu, Max-margin majority voting for learning from crowds, Advances in Neural Information Processing Systems, pp.1621-1629, 2015.
DOI : 10.1109/tpami.2018.2860987

L. Tran-thanh, Efficient Budget Allocation with Accuracy Guarantees for Crowdsourcing Classification Tasks, pp.978-979, 2013.

. Amazon-mechanical and . Turk,

. Twitter and . Url,

C. Van-pelt and A. Sorokin, Designing a scalable crowdsourcing platform, Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, pp.765-766, 2012.

L. Von-ahn and L. Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, pp.319-326, 2004.

. Luis-von-ahn, recaptcha: Human-based character recognition via web security measures, vol.321, pp.1465-1468, 2008.

J. Wang, G. Panagiotis, F. Ipeirotis, and . Provost, Quality-based pricing for crowdsourced workers, p.2013

P. Welinder, The multidimensional wisdom of crowds, Advances in neural information processing systems, vol.57, pp.2424-2432, 2010.

J. Weng, Twitterrank: finding topic-sensitive influential twitterers, Proceedings of the third ACM international conference on Web search and data mining, pp.261-270, 2010.

J. Whitehill, Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise, Conference on Neural Information Processing Systems, vol.6, pp.12-14, 2009.

. Mark-e-whiting, Crowd guilds: Worker-led reputation and feedback on crowdsourcing platforms, vol.17, 2016.

. Wikipedia and . Url,

. Wikipedia and . Wikipedia, Contributing to Wikipedia. accessed 05-03-2018

. Mark-d-wilkinson, The FAIR Guiding Principles for scientific data management and stewardship, Scientific data, vol.3, 2016.

D. Yang, Incentive mechanisms for crowdsensing: Crowdsourcing with smartphones, IEEE/ACM Transactions on Networking (TON), vol.24, pp.1732-1744, 2016.
DOI : 10.1109/tnet.2015.2421897

K. Yang, Security and privacy in mobile crowdsourcing networks: challenges and opportunities, IEEE communications magazine, vol.53, pp.75-81, 2015.
DOI : 10.1109/mcom.2015.7180511

Y. Bin-ye, L. Wang, and . Liu, Crowd trust: A context-aware trust model for worker selection in crowdsourcing environments, 2015 IEEE International Conference on Web Services (ICWS). IEEE. 2015, vol.30, pp.121-128

M. Yin and Y. Chen, Bonus or Not? Learn to Reward in Crowdsourcing, International Joint Conference on Artificial Intelligence, pp.201-208, 2015.

M. Yuen, I. King, and K. Leung, TaskRec: A Task Recommendation Framework in Crowdsourcing Systems, Neural Processing Letters, vol.41, pp.223-238, 2015.
DOI : 10.1007/s11063-014-9343-z

. Zelda-b-zabinsky, Random search algorithms, Wiley Encyclopedia of Operations Research and Management Science, p.66, 2010.

F. Omar, C. Zaidan, and . Callison-burch, Crowdsourcing Translation: Professional Quality from Non-professionals, Oregon, pp.978-979, 2011.

W. Zhang, T. Yoshida, and X. Tang, TFIDF, LSI and multi-word in information retrieval and text categorization, IEEE SMC, p.42, 2008.

Y. Zhang, Spectral methods meet EM: A provably optimal algorithm for crowdsourcing, Advances in neural information processing systems, pp.1260-1268, 2014.

S. Zhong, Efficient online spherical k-means clustering, vol.5, p.75, 2005.

D. Zhou, Aggregating ordinal labels from crowds by minimax conditional entropy, International conference on machine learning, pp.262-270, 2014.

D. Zhou, Learning from the wisdom of crowds by minimax entropy, Advances in neural information processing systems, vol.57, p.13, 2012.