A. Sackl, M. Seufert, and T. Hoßfeld, Asking costs little? The impact of tasks in video QoE studies on user behavior and user ratings, 4th International Workshop on Perceptual Quality of Systems, 2013.

B. Naderi, Asking costs little? The impact of tasks in video QoE studies on user behavior and user ratings, 3rd International ACM workshop on Crowdsourcing for Multimedia, 2014.

B. Naderi, Crowdee: Mobile Crowdsourcing Micro-Task Platform for Celebrating the Diversity of Languages, Fifteenth Annual Conference of the International Speech Communication Association, 2014.

B. Gardlo, Microworkers vs. facebook: The impact of crowdsourcing platform choice on experimental results, 2012 Fourth International Workshop on Quality of Multimedia Experience, 2012.
DOI : 10.1109/QoMEX.2012.6263885

B. Gardlo, M. Ries, and T. Hoßfeld, Impact of screening technique on crowdsourcing QoE assessments, 22nd International Conference Radioelektronika. IEEE. Brno, Czech Republic, 2012.

B. Gardlo, Crowdsourcing 2.0: Enhancing execution speed and reliability of web-based QoE testing, 2014 IEEE International Conference on Communications (ICC), 2014.
DOI : 10.1109/ICC.2014.6883463

C. Keimel, Qualitycrowda framework for crowd-based quality evaluation, Picture Coding Symposium, 2012.

C. Keimel, Video quality evaluation in the cloud, 2012 19th International Packet Video Workshop (PV), 2012.
DOI : 10.1109/PV.2012.6229729

C. Keimel, J. Habigt, and K. Diepold, Challenges in crowd-based video quality assessment, 2012 Fourth International Workshop on Quality of Multimedia Experience, 2012.
DOI : 10.1109/QoMEX.2012.6263866

F. Mazza, M. P. , D. Silva, and P. L. Callet, Would you hire me? Selfie portrait images perception in a recruitment context, IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, 2014.
URL : https://hal.archives-ouvertes.fr/hal-00952704

I. Hupont, Is affective crowdsourcing reliable?, 2014 IEEE Fifth International Conference on Communications and Electronics (ICCE), 2014.
DOI : 10.1109/CCE.2014.6916757

J. Redi and I. Povoa, Crowdsourcing for Rating Image Aesthetic Appeal, Proceedings of the 2014 International ACM Workshop on Crowdsourcing for Multimedia, CrowdMM '14, 2014.
DOI : 10.1145/2660114.2660118

J. Redi, Crowdsourcing-based multimedia subjective evaluations, Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia, CrowdMM '13, 2013.
DOI : 10.1145/2506364.2506368
URL : https://hal.archives-ouvertes.fr/hal-00952693

M. Varela, Increasing Payments in Crowdsourcing: Don't look a gift horse in the mouth, 4th International Workshop on Perceptual Quality of Systems, 2013.

M. Hirth, Predicting result quality in Crowdsourcing using application layer monitoring, 2014 IEEE Fifth International Conference on Communications and Electronics (ICCE), 2014.
DOI : 10.1109/CCE.2014.6916756

M. Hirth, T. Hoßfeld, and P. Tran-gia, Analyzing costs and accuracy of validation mechanisms for crowdsourcing platforms, Mathematical and Computer Modelling, vol.57, issue.11-12, pp.2918-2932, 2013.
DOI : 10.1016/j.mcm.2012.01.006

M. Hirth, T. Hoßfeld, and P. Tran-gia, Anatomy of a crowdsourcing platformusing the example of microworkers. com, Fifth International Conference On Innovative Mobile and Internet Services in Ubiquitous Computing, 2011.

M. Hirth, T. Hoßfeld, and P. Tran-gia, Cost-Optimal Validation Mechanisms and Cheat-Detection for Crowdsourcing Platforms, 2011 Fifth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing, 2011.
DOI : 10.1109/IMIS.2011.91

M. Seufert, “To pool or not to pool”: A comparison of temporal pooling methods for HTTP adaptive video streaming, 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX), 2013.
DOI : 10.1109/QoMEX.2013.6603210

P. Korshunov, The effect of HDR images on privacy: crowdsourcing evaluation, SPIE Photonics Europe 2014, Optics, Photonics and Digital Technologies for Multimedia Applications, 2014.

P. Amrehn, Need for speed? On quality of experience for file storage services, 4th International Workshop on Perceptual Quality of Systems, 2013.

P. Tran-gia, Crowdsourcing and its Impact on Future Internet Usage, it-Information Technology Methoden und innovative Anwendungen der Informatik und Informationstechnik 55, pp.139-145, 2013.
DOI : 10.1524/itit.2013.1005

S. Oechsner, Visions and Challenges for Sensor Network Collaboration in the Cloud, 2014 Eighth International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing, 2011.
DOI : 10.1109/IMIS.2014.3

T. Hoßfeld, Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing, IEEE Transactions on Multimedia 16, pp.541-558, 2014.
DOI : 10.1109/TMM.2013.2291663

T. Hoßfeld and C. Keimel, Crowdsourcing in QoE Evaluation Quality of Experience: Advanced Concepts, Applications and Methods, 2014.

T. Hoßfeld and C. Timmerer, Quality of Experience Assessment using Crowdsourcing, IEEE COMSOC MMTC R-Letter, vol.5, 2014.

T. Hoßfeld, Survey of Web-based Crowdsourcing Frameworks for Subjective Quality Assessment, 16th International Workshop on Multimedia Signal Processing, 2014.

T. Hoßfeld, M. Hirth, and P. Tran-gia, Modeling of crowdsourcing platforms and granularity of work organization in future internet International Teletraffic Congress, 23rd International Teletraffic Congress, 2011.

T. Hoßfeld, Assessing effect sizes of influence factors towards a QoE model for HTTP adaptive streaming, 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX), 2014.
DOI : 10.1109/QoMEX.2014.6982305

T. Hoßfeld, Quantification of YouTube QoE via Crowdsourcing, 2011 IEEE International Symposium on Multimedia, 2011.
DOI : 10.1109/ISM.2011.87

T. Hoßfeld, Initial delay vs. interruptions: Between the devil and the deep blue sea, 2012 Fourth International Workshop on Quality of Multimedia Experience, 2012.
DOI : 10.1109/QoMEX.2012.6263849

T. Hoßfeld, On Training the Crowd for Subjective Quality Studies, 2014.

V. Burger, Increasing the Coverage of Vantage Points in Distributed Active Network Measurements by Crowdsourcing, Measurement, Modelling, and Evaluation of Computing Systems and Dependability and Fault Tolerance (MMB & DFT 2014, 2014.
DOI : 10.1007/978-3-319-05359-2_11

T. Hoßfeld, P. Tran-gia, and M. Vucovic, Crowdsourcing, Informatik-Spektrum, vol.35, issue.3, 2013.
DOI : 10.1007/s00287-012-0610-y

M. Rerabek, Evaluation of privacy in high dynamic range video sequences, SPIE Optical Engineering + Applications. International Society for Optics and Photonics, 2014.

P. Hanhart, P. Korshunov, and T. Ebrahimi, Crowdsourcing evaluation of high dynamic range image compression, SPIE Optical Engineering + Applications. International Society for Optics and Photonics, 2014.

P. Hanhart, P. Korshunov, and T. Ebrahimi, Crowd-based quality assessment of multiview video plus depth coding, 2014 IEEE International Conference on Image Processing (ICIP), 2014.
DOI : 10.1109/ICIP.2014.7025149

P. Korshunov, S. Cai, and T. Ebrahimi, Crowdsourcing approach for evaluation of privacy filters in video surveillance, Proceedings of the ACM multimedia 2012 workshop on Crowdsourcing for multimedia, CrowdMM '12, 2012.
DOI : 10.1145/2390803.2390817

B. Gardlo, Quality of experience evaluation methodology via crowdsourcing, 2012.

P. Lebreton, Bridging the Gap Between Eye Tracking and Crowdsourcing Human Vision and Electronic Imaging XX, Proceedings of the SPIE 9394, 2015.

B. Gardlo, S. Egger, and T. Hoßfeld, Does Scale-Design matter for assessing Video QoE through Crowdsourcing? Tech. rep. FTW, 2014.

O. Alonso, E. Daniel, B. Rose, and . Stewart, Crowdsourcing for relevance evaluation, ACM SIGIR Forum, vol.42, issue.2, pp.9-15, 2008.
DOI : 10.1145/1480506.1480508

S. Julie and . Downs, Are your participants gaming the system?: screening mechanical turk workers, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp.2399-2402, 2010.

P. Hsueh, P. Melville, and V. Sindhwani, Data quality from crowdsourcing, Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, HLT '09, pp.27-35, 2009.
DOI : 10.3115/1564131.1564137

Q. Xu, HodgeRank on Random Graphs for Subjective Video Quality Assessment, IEEE Transactions on Multimedia, vol.14, issue.3, pp.844-857, 2012.
DOI : 10.1109/TMM.2012.2190924

K. Chen, A crowdsourceable QoE evaluation framework for multimedia content, Proceedings of the seventeen ACM international conference on Multimedia, MM '09, 2009.
DOI : 10.1145/1631272.1631339

. European-parliament-directive, 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data and on the free movement of such data, In: Official Journal of the EC, vol.236, 1995.

C. Lilly, M. Irani, and . Silberman, Turkopticon: Interrupting worker invisibility in amazon mechanical turk, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2013.

R. Huib-de, Cognitive issues in image quality measurement, Journal of Electronic Imaging, vol.10, issue.1, pp.47-55, 2001.

Y. Pitrey, Aligning subjective tests using a low cost common set, QoE for Multimedia Content Sharing, 2011.
URL : https://hal.archives-ouvertes.fr/hal-00608310

A. Eichhorn, P. Ni, and R. Eg, Randomised pair comparison, Proceedings of the 20th international workshop on Network and operating systems support for digital audio and video, NOSSDAV '10, 2010.
DOI : 10.1145/1806565.1806582

E. Peter, Z. Rossi, G. M. Gilula, and . Allenby, Overcoming scale usage heterogeneity: A Bayesian hierarchical approach, Journal of the American Statistical Association, vol.96453, pp.20-31, 2001.

L. Bronwen, . Jones, R. Pamela, and . Mcmanus, Graphic scaling of qualitative terms, SMPTE journal, vol.9511, pp.1166-1171, 1986.

M. Virtanen, M. Gleiss, and . Goldstein, On the use of evaluative category scales in telecommunications, Human Factors in Telecommunications, 1995.

C. Wu, Crowdsourcing Multimedia QoE Evaluation: A Trusted Framework, IEEE transactions on multimedia 15, pp.1121-1137, 2013.
DOI : 10.1109/TMM.2013.2241043

J. Antin and A. Shaw, Social desirability bias and self-reports of motivation, Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, CHI '12, 2012.
DOI : 10.1145/2207676.2208699

G. Panagiotis and . Ipeirotis, Demographics of mechanical turk, 2010.