. P. Itu-t-rec, Parametric bitstream-based quality assessment of progressive download and adaptive audiovisual streaming services over reliable transport, ITU-T Rec. P.1203, 1203.

, Ericsson Mobility Report, 2018.

T. Hoßfeld, M. Seufert, M. Hirth, T. Zinner, P. Tran-gia et al., Quantification of youtube qoe via crowdsourcing, Multimedia (ISM), 2011 IEEE International Symposium on, pp.494-499, 2011.

G. Dimopoulos and I. Leontiadis, Pere Barlet-Ros, and Konstantina Papagiannaki. Measuring video qoe from encrypted traffic, Proceedings of the, 2016.

, Internet Measurement Conference, pp.513-526, 2016.

K. P. Ricky, . Mok, W. W. Edmond, R. K. Chan, and . Chang, Measuring the quality of experience of http video streaming, Integrated Network Management, pp.485-492, 2011.

Z. Duanmu, K. Zeng, K. Ma, A. Rehman, and Z. Wang, A quality-of-experience index for streaming video, IEEE Journal of Selected Topics in Signal Processing, vol.11, issue.1, pp.154-166, 2017.

Ö. Alay, A. Lutu, M. Peón-quirós, V. Mancuso, T. Hirsch et al., Experience: An open platform for experimentation with commercial mobile broadband networks, Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking, MobiCom '17, pp.70-78, 2017.

A. Schwind, M. Seufert, Ö. Alay, P. Casas, P. Tran-gia et al., Concept and implementation of video qoe measurements in a mobile broadband testbed, Network Traffic Measurement and Analysis Conference (TMA), pp.1-6, 2017.

A. K. Moorthy, K. Seshadrinathan, R. Soundararajan, and A. C. Bovik, Wireless video quality assessment: A study of subjective scores and objective algorithms, IEEE Transactions on Circuits and Systems for Video Technology, vol.20, pp.587-599, 2010.

T. Hoßfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt et al., Best practices for qoe crowdtesting: Qoe assessment with crowdsourcing

, IEEE Transactions on Multimedia, vol.16, issue.2, pp.541-558, 2014.

C. Wu, K. Chen, Y. Chang, and C. Lei, Crowdsourcing multimedia qoe evaluation: A trusted framework, IEEE Transactions on Multimedia, vol.15, issue.5, pp.1121-1137, 2013.

C. Kuan-ta-chen, C. Chang, Y. Wu, C. Chang, and . Lei, Quadrant of euphoria: A crowdsourcing platform for qoe assessment

, Mag. of Global Internetwkg, vol.24, issue.2, pp.28-35, 2010.

K. Chen, C. Wu, Y. Chang, and C. Lei, A crowdsourceable qoe evaluation framework for multimedia content, Proceedings of the 17th ACM International Conference on Multimedia, MM '09, pp.491-500, 2009.

U. Gadiraju, S. Möller, M. Nöllenburg, D. Saupe, S. Egger-lampl et al., Crowdsourcing Versus the Laboratory: Towards Human-Centered Experiments Using the Crowd, pp.6-26, 2017.

T. Spetebroot, S. Afra, N. Aguilera, D. Saucez, and C. Barakat, From networklevel measurements to expected quality of experience: The skype use case, Measurements Networking (M & N), pp.1-6, 2015.
URL : https://hal.archives-ouvertes.fr/hal-01071373

O. Belmoukadam, T. Spetebroot, and C. Barakat, Acqua: A user friendly platform for lightweight network monitoring and qoe forecasting, 2019 22nd Conference on Innovation in Clouds, Internet and Networks and Workshops (ICIN), pp.88-93, 2019.
URL : https://hal.archives-ouvertes.fr/hal-01982749

, Meteor: Free internet speed & app performance test, 2018.

P. Schmitt, F. Bronzino, R. Teixeira, T. Chattopadhyay, and N. Feamster, Enhancing transparency: Internet video quality inference from network traffic, TPRC 46: The 46th Research Conference on Communication, Information and Internet Policy, 2018.
URL : https://hal.archives-ouvertes.fr/hal-02074828

T. Hoßfeld, R. Schatz, E. Biersack, and L. Plissonneau, Internet Video Delivery in YouTube: From Traffic Measurements to Quality of Experience, pp.264-301, 2013.

. Rtr-netz-open and . Dataset, , 2017.

. Mobiperf, , 2018.

M. H. Mazhar and Z. Shafiq, Real-time video quality of experience monitoring for https and quic, IEEE INFOCOM 2018 -IEEE Conference on Computer Communications, 2018.

I. Orsolic, D. Pevec, M. Suznjevic, and L. Skorin-kapov, A machine learning approach to classifying youtube qoe based on encrypted network traffic, Multimedia Tools and Applications, vol.76, 2017.

A. Muhammad-jawad-khokhar, T. Saber, C. Spetebroot, and . Barakat, On active sampling of controlled experiments for qoe modeling

, Proceedings of the Workshop on QoE-based Analysis and Management of Data Communication Networks, Internet QoE '17, pp.31-36, 2017.

M. J. Khokhar, T. Spetebroot, and C. Barakat, An online sampling approach for controlled experimentation and qoe modeling, 2018 IEEE International Conference on Communications (ICC), pp.1-6, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01677378

A. Muhammad-jawad-khokhar, T. Saber, C. Spetebroot, and . Barakat, An intelligent sampling framework for controlled experimentation and qoe modeling, Computer Networks, vol.147, pp.246-261, 2018.

M. Khokhar, T. Spetebroot, and C. Barakat, A methodology for performance benchmarking of mobile networks for internet video streaming, Proceedings of the 21st ACM International Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, MSWIM '18, pp.217-225, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01855264

T. Muhammad-jawad-khokhar, C. Ehlinger, and . Barakat, From Network Traffic Measurements to QoE for Internet Video, IFIP Networking Conference, 2019.

Y. Qoe,

D. Code, , 2018.

H. Schulzrinne, A. Rao, and R. Lanphier, Real time streaming protocol (rtsp), 1998.

H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, Rtp: A transport protocol for real-time applications, 2003.

P. Juluri, V. Tamarapalli, and D. Medhi, Measurement of quality of experience of video-on-demand services: A survey, IEEE Communications Surveys Tutorials, vol.18, issue.1, pp.401-418, 2016.

C. G. Bampis and A. C. Bovik, Learning to predict streaming video qoe: Distortions, rebuffering and memory, 2017.

Y. Sani, A. Mauthe, and C. Edwards, Adaptive bitrate selection: A survey, IEEE Communications Surveys Tutorials, vol.19, issue.4, pp.2985-3014, 2017.

S. Akhshabi, A. C. Begen, and C. Dovrolis, An experimental evaluation of rate-adaptation algorithms in adaptive streaming over http, Proceedings of the Second Annual ACM Conference on Multimedia Systems, MMSys '11, pp.157-168, 2011.

T. Hossfeld, S. Egger, R. Schatz, M. Fiedler, K. Masuch et al., Initial delay vs. interruptions: Between the devil and the deep blue sea, 2012 Fourth International Workshop on Quality of Multimedia Experience, pp.1-6, 2012.

S. , S. Krishnan, and R. K. Sitaraman, Video stream quality impacts viewer behavior: Inferring causality using quasi-experimental designs, Proceedings of the 2012 Internet Measurement Conference, IMC '12, pp.211-224, 2012.

K. D. Singh, Y. Hadjadj-aoul, and G. Rubino, Quality of experience estimation for adaptive http/tcp video streaming using h.264/avc, 2012 IEEE Consumer Communications and Networking Conference (CCNC), pp.127-131, 2012.
URL : https://hal.archives-ouvertes.fr/hal-00713723

I. Sodagar, The mpeg-dash standard for multimedia streaming over the internet, IEEE MultiMedia, vol.18, issue.4, pp.62-67, 2011.

, IIS smooth streaming transport protocol, 2014.

, Adobe HTTP Dynamic Streaming, 2014.

, HTTP Live Streaming, Informational In-ternet Draft, 2011.

T. Huang, R. Johari, N. Mckeown, M. Trunnell, and M. Watson, A buffer-based approach to rate adaptation: Evidence from a large video streaming service, Proceedings of the 2014 ACM Conference on SIGCOMM, SIGCOMM '14, pp.187-198, 2014.

K. Spiteri, R. Urgaonkar, and R. K. Sitaraman, Bola: Near-optimal bitrate adaptation for online videos, IEEE INFOCOM 2016 -The 35th Annual IEEE International Conference on Computer Communications, pp.1-9, 2016.

J. Jiang, V. Sekar, and H. Zhang, Improving fairness, efficiency, and stability in http-based adaptive video streaming with festive, IEEE/ACM Trans. Netw, vol.22, issue.1, pp.326-340, 2014.

G. Tian and Y. Liu, Towards agile and smooth video adaptation in dynamic http streaming, Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies, CoNEXT '12, pp.109-120

N. Y. York and . Usa, , 2012.

X. Yin, A. Jindal, V. Sekar, and B. Sinopoli, A control-theoretic approach for dynamic adaptive video streaming over http, Proceedings of ACM SIGCOMM, pp.325-338, 2015.

Z. Akhtar, Y. S. Nam, R. Govindan, S. Rao, J. Chen et al., Oboe: Autotuning video abr algorithms to network conditions, Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication, SIGCOMM '18, pp.44-58, 2018.

H. Mao, R. Netravali, and M. Alizadeh, Neural adaptive video streaming with pensieve, Proceedings of the Conference of the ACM Special Interest Group on Data Communication, SIGCOMM '17, pp.197-210, 2017.

M. Gadaleta, F. Chiariotti, M. Rossi, and A. Zanella, D-dash: A deep qlearning framework for dash video streaming, IEEE Transactions on Cognitive Communications and Networking, vol.3, issue.4, pp.703-718, 2017.

P. Brooks and B. Hestnes, User measures of quality of experience: why being objective and quantitative is important, IEEE Network, vol.24, issue.2, pp.8-13, 2010.

M. Alreshoodi and J. Woods, Survey on qoe qos correlation models for multimedia services, International Journal of Distributed and Parallel systems, vol.4, 2013.

A. K. Moorthy, L. K. Choi, A. C. Bovik, and G. De-veciana, Video quality assessment on mobile devices: Subjective, behavioral and objective studies, IEEE Journal of Selected Topics in Signal Processing, vol.6, issue.6, pp.652-671, 2012.

. Itu-t, Subjective video quality assessment methods for multimedia applications, ITU-T Recommendation P.910, 2008.

S. Aroussi and A. Mellouk, Survey on machine learning-based qoe-qos correlation models, 2014 International Conference on Computing, Management and Telecommunications (ComManTel), pp.200-204, 2014.

A. Balachandran, V. Sekar, A. Akella, S. Seshan, I. Stoica et al., Developing a predictive model of quality of experience for internet video, Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, pp.339-350, 2013.

Y. Qi and M. Dai, The effect of frame freezing and frame skipping on video quality, 2006 International Conference on Intelligent Information Hiding and Multimedia, pp.423-426, 2006.

M. Tahir-nawaz-minhas and . Fiedler, Impact of disturbance locations on video quality of experience, Workshop: Quality of Experience for Multimedia Content Sharing, 2011.

Q. Huynh-thu and M. Ghanbari, Temporal aspect of perceived quality in mobile video broadcasting, IEEE Transactions on Broadcasting, vol.54, issue.3, pp.641-651, 2008.

R. Serral-gracià, E. Cerqueira, M. Curado, M. Yannuzzi, E. Monteiro et al., An overview of quality of experience measurement challenges for video applications in ip networks, Wired/Wireless Internet Communications, pp.252-263, 2010.

A. C. Zhou-wang, H. R. Bovik, E. P. Sheikh, and . Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing, vol.13, issue.4, pp.600-612, 2004.

, Toward A Practical Perceptual Video Quality Metric, 2016.

P. Eero, . Simoncelli-zhou, and . Wang, Reduced-reference image quality assessment using a wavelet-domain natural image statistic model, 2005.

W. Lin and C. Kuo, Perceptual visual quality metrics: A survey, Journal of Visual Communication and Image Representation, vol.22, issue.4, pp.297-312, 2011.

B. Lewcio, B. Belmudez, T. Enghardt, and S. Möller, On the way to high-quality video calls in future mobile networks, 2011 Third International Workshop on Quality of Multimedia Experience, pp.43-48, 2011.

B. Lewcio, B. Belmudez, A. Mehmood, M. Wältermann, and S. Möller, Video quality in next generation mobile networks -perception of time-varying transmission, 2011 IEEE International Workshop Technical Committee on Communications Quality and Reliability (CQR), pp.1-6, 2011.

M. Zink, J. Schmitt, and R. Steinmetz, Layer-encoded video in scalable adaptive streaming, IEEE Transactions on Multimedia, vol.7, issue.1, pp.75-84, 2005.

P. Ni, R. Eg, A. Eichhorn, C. Griwodz, and P. Halvorsen, Flicker effects in adaptive video streaming to handheld devices, Proceedings of the 19th ACM International Conference on Multimedia, MM '11, pp.463-472, 2011.

S. Khirman and P. Henriksen, Relationship between quality-of-service and qualityof-experience for public internet service, 3rd Passive Active Measurement Workshop, 2002.

M. Fiedler, T. Hossfeld, and P. Tran-gia, A generic quantitative relationship between quality of experience and quality of service, IEEE Network, vol.24, issue.2, pp.36-41, 2010.

A. Youtube, , 2019.

. Webrequest, , 2018.

. Microworkers, , 2019.

F. Wamser, M. Seufert, P. Casas, R. Irmer, P. Tran-gia et al., Yomoapp: A tool for analyzing qoe of youtube http adaptive streaming in mobile networks, 2015 European Conference on Networks and Communications (EuCNC), pp.239-243, 2015.

M. Seufert, N. Wehner, F. Wamser, P. Casas, A. D'alconzo et al.,

, Unsupervised qoe field study for mobile youtube video streaming with yomoapp, QoMEX, pp.1-6, 2017.

P. Juluri, L. Plissonneau, and D. Medhi, Pytomo: A tool for analyzing playback quality of youtube videos, 23rd International Teletraffic Congress (ITC), pp.304-305, 2011.

C. Gutterman, K. Guo, S. Arora, X. Wang, L. Wu et al., Requet: Real-time qoe detection for encrypted youtube traffic, Proc. ACM MMSys'19, 2019.

X. Liu, F. Dobrian, H. Milner, J. Jiang, V. Sekar et al., A case for a coordinated internet video control plane, Proceedings of the ACM SIGCOMM 2012 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, SIGCOMM '12, pp.359-370, 2012.

F. Dobrian, V. Sekar, A. Awan, I. Stoica, D. Joseph et al., Understanding the impact of video quality on user engagement, SIGCOMM CCR, vol.41, issue.4, 2011.

L. Yitong, S. Yun, M. Yinian, L. Jing, L. Qi et al., A study on quality of experience for adaptive streaming service, 2013 IEEE International Conference on Communications Workshops (ICC), pp.682-686, 2013.

M. Katsarakis, R. C. Teixeira, M. Papadopouli, and V. Christophides, Towards a causal analysis of video qoe from network and application qos, Proceedings of the Internet QoE'16, pp.31-36, 2016.
URL : https://hal.archives-ouvertes.fr/hal-01338726

T. Hoßfeld, R. Schatz, and U. R. Krieger, Qoe of youtube video streaming for current internet transport protocols

. Krieger, Measurement, Modelling, and Evaluation of Computing Systems and Dependability and Fault Tolerance, pp.136-150, 2014.

A. Nikravesh, D. K. Hong, Q. A. Chen, H. V. Madhyastha, and Z. Mao, Qoe inference without application control, Proceedings of the 2016 Workshop on QoE-based Analysis and Management of Data Communication Networks, Internet-QoE '16, pp.19-24, 2016.

T. Mangla, E. Halepovic, M. Ammar, and E. Zegura, emimic: Estimating httpbased video qoe metrics from encrypted network traffic, Network Traffic Measurement and Analysis Conference (TMA), pp.1-8, 2018.

. Muhammad-zubair, J. Shafiq, L. Erman, A. X. Ji, J. Liu et al., Understanding the impact of network dynamics on mobile video user engagement, The 2014 ACM Int'l Conference on Measurement and Modeling of Computer Systems, pp.367-379, 2014.

J. Jiang, V. Sekar, H. Milner, D. Shepherd, I. Stoica et al., CFA: A practical prediction system for video qoe optimization, 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI 16), 2016.

B. Settles, Active learning literature survey, 2010.

B. C. Wallace, K. Small, C. E. Brodley, and T. A. Trikalinos, Active learning for biomedical citation screening, Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.173-182, 2010.

D. Angluin, Queries and concept learning, Machine Learning, vol.2, pp.319-342, 1988.

F. Pedregosa, Scikit-learn: Machine learning in python, Journal of Machine Learning Research, 2011.
URL : https://hal.archives-ouvertes.fr/hal-00650905

. Ting-fan, C. Wu, R. C. Lin, and . Weng, Probability estimates for multiclass classification by pairwise coupling, J. Mach. Learn. Res, vol.5, 2004.

L. Rizzo, Dummynet: A simple approach to the evaluation of network protocols. SIGCOMM Comput, Commun. Rev, vol.27, issue.1, pp.31-41, 1997.

, Subjective audiovisual quality assessment methods for multimedia applications, ITU-T Recommendation P.911, 1998.

V. Aggarwal, E. Halepovic, J. Pang, S. Venkataraman, and H. Yan, Prometheus: Toward quality-of-experience estimation for mobile apps from passive network measurements, Proceedings of the 15th Workshop on Mobile Computing Systems and Applications, HotMobile '14, vol.18, pp.1-18, 2014.

J. Zhu, H. Wang, E. Hovy, and M. Ma, Confidence-based stopping criteria for active learning for data annotation, ACM Trans. Speech Lang. Process, vol.6, issue.3, p.24, 2010.

, Raw Data -Measuring Broadband America, 2016.

V. Menkovski, G. Exarchakos, and A. Liotta, Tackling the sheer scale of subjective qoe, Mobile Multimedia Communications, pp.1-15, 2012.

P. Ye and D. Doermann, Active sampling for subjective image quality assessment, 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp.4249-4256, 2014.

R. Hashemian, N. Carlsson, D. Krishnamurthy, and M. Arlitt, Iris: Iterative and intelligent experiment selection, Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, ICPE '17, pp.143-154, 2017.

I. Zliobaite, A. Bifet, B. Pfahringer, and G. Holmes, Active learning with drifting streaming data, IEEE Transactions on Neural Networks and Learning Systems, vol.25, issue.1, pp.27-39, 2014.

H. Chang, C. Hsu, T. Hobfeld, and K. Chen, Active learning for crowdsourced qoe modeling, IEEE Transactions on Multimedia, pp.1-1, 2018.

P. Casas, M. Seufert, N. Wehner, A. Schwind, and F. Wamser, Enhancing machine learning based qoe prediction by ensemble models, 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS), pp.1642-1647, 2018.

V. Vasilev, J. Leguay, S. Paris, L. Maggi, and M. Debbah, Predicting qoe factors with machine learning, 2018 IEEE International Conference on Communications (ICC), pp.1-6, 2018.
URL : https://hal.archives-ouvertes.fr/hal-01947195

J. D. Vriendt, D. D. Vleeschauwer, and D. C. Robinson, Qoe model for video delivered over an lte network using http adaptive streaming, Bell Labs Technical Journal, vol.18, issue.4, pp.45-62, 2014.

F. Wamser, P. Casas, M. Seufert, C. Moldovan, P. Tran-gia et al., Modeling the youtube stack: From packets to quality of experience, Computer Networks, vol.109, 2016.

. Vp9-codec, , 2018.

, Linux Traffic Control, 2018.

I. Orsolic, D. Pevec, M. Suznjevic, and L. Skorin-kapov, Youtube qoe estimation based on the analysis of encrypted network traffic using machine learning, IEEE Globecom Workshops, 2016.

X. Che, B. Ip, and L. Lin, A survey of current youtube video characteristics, IEEE MultiMedia, vol.22, issue.2, pp.56-63, 2015.

A. Nikravesh, D. R. Choffnes, E. Katz-bassett, Z. M. Mao, and M. Welsh, Mobile network performance from user devices: A longitudinal, multidimensional analysis, Proceedings of PAM 2014, pp.12-22, 2014.

S. Rosen, H. Yao, A. Nikravesh, Y. Jia, D. Choffnes et al., Demo: Mapping global mobile performance trends with mobilyzer and mobiperf, Proceedings of MobiSys '14, pp.353-353, 2014.

. Grid5000, , 2018.

, AWS EC2, 2018.

W. Robitza, HTTP Adaptive Streaming QoE Estimation with ITU-T Rec. P.1203 -Open Databases and Software, 9th ACM Multimedia Systems Conference, 2018.

. Itu-p, 1203 Python implementation, 2018.

, Ericsson QoE report, 2017.

, Python Scikit-learn library, 2018.

V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap et al., Asynchronous methods for deep reinforcement learning, Proceedings of The 33rd International Conference on Machine Learning, vol.48, pp.20-22, 2016.

R. S. Sutton, D. Mcallester, S. Singh, and Y. Mansour, Policy gradient methods for reinforcement learning with function approximation, Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS'99, pp.1057-1063, 1999.

, Blockchain as a Service