Survey on QoE Assessment Approach for Network Service
Open accessAbstract
With the development of converged all-IP networks, new network services-supporting, among others, image, audio, and video-with high diversity emerge, which brings huge pressure and fierce competition to service providers because of increasing choices for users. Assessing end user's quality of experience (QoE) has become one of the main issues to meet high user requirements and satisfaction under any circumstances, which means receiving any service, anytime, anywhere, on any device, by any media and networking technology, across any operator domain. In this paper, we provide an overview of the field of QoE assessment approaches including three aspects: subjective assessment, objective assessment, and hybrid assessment. The discussion includes the definition, influence factors, and quantification approaches for QoE. Then, these three assessment methodologies are surveyed and compared across a number of factors. This paper also introduces challenges and opportunities in the area of QoE assessment.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing surveys on QoE in IoT focus on network metrics to assess QoE.
“Existing surveys on QoE in IoT such as [16][17][18] focus on network metrics to assess QoE.”
From QoE in IoT: a vision, survey and future directions (2021) · cites this paper as [16]
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · see citing works
Multiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253 · cited 3× · see citing works
“• MultiScale-SSIM (MS-SSIM) [49] is an extension of single-scale SSIM, originally proposed for still image and later extended to video.”
MethodInfluentialImage Quality Metrics: PSNR vs. SSIM
Hore, A., & Ziou, D. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.579 · Cited by 4,943 · see citing works
Image information and visual quality
Sheikh, H. R., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.859378 · Cited by 4,104 · see citing works
Scope of validity of PSNR in image/video quality assessment
Huynh-Thu, Q., & Ghanbari, M. (2008) · Electronics Letters · doi:10.1049/el:20080522 · Cited by 2,959 · see citing works
A new standardized method for objectively measuring video quality
Pinson, M. H., & Wolf, S. (2004) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2004.834028 · Cited by 1,182 · cited 2× · see citing works
“• Video Quality Metric (VQM) [37], [54] provides standardized and non-standardized approaches to measure”
MethodStudy of Subjective and Objective Quality Assessment of Video
Seshadrinathan, K., Soundararajan, R., Bovik, A. C., & Cormack, L. K. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2042111 · Cited by 1,160 · see citing works
VSNR: A Wavelet-Based Visual Signal-to-Noise Ratio for Natural Images
Chandler, D. M., & Hemami, S. S. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2007.901820 · Cited by 1,147 · see citing works
Why is image quality assessment so difficult?
Wang, Z., Bovik, A. C., & Lu, L. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5745362 · Cited by 787 · see citing works
Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos
Seshadrinathan, K., & Bovik, A. C. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2034992 · Cited by 708 · cited 1× · see citing works
“[60] detects video distortion not separately in space and time domain, but rather in space-time domain through computing motion trajectories of the object in video and making spatio-temporal distortion assessments.”
BackgroundQualinet White Paper on Definitions of Quality of Experience
Callet, P. L., Möller, S., Perkis, A., Brunnstrom, K. E., Beker, S., Moor, K. D., Dooms, A., Egger, S., Garcia, M.-N., Hoßfeld, T., et al. (2013) · HAL · Cited by 692 · see citing works
Objective Video Quality Assessment Methods: A Classification, Review, and Performance Comparison
Chikkerur, S., Sundaram, V., Martin Reisslein, B., & Karam, L. J. (2011) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2011.2104671 · Cited by 613 · see citing works
Teo, P. C., & Heeger, D. J. (2002) · Proceedings of 1st International Conference on Image Processing · doi:10.1109/icip.1994.413502 · Cited by 471 · see citing works
From QoS to QoE: A Tutorial on Video Quality Assessment
Chen, Y., Wu, K., & Zhang, Q. (2014) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2014.2363139 · Cited by 342 · see citing works
Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing
Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K. J., & Tran‐Gia, P. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2291663 · Cited by 283 · see citing works
User measures of quality of experience: why being objective and quantitative is important
Brooks, P., & Hestnes, B. (2010) · IEEE Network · doi:10.1109/mnet.2010.5430138 · Cited by 268 · see citing works
Measurement of Quality of Experience of Video-on-Demand Services: A Survey
Juluri, P., Tamarapalli, V., & Medhi, D. (2016) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2015.2401424 · Cited by 229 · see citing works
Visual-PSNR measure of image quality
Tanchenko, A. (2014) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2014.01.008 · Cited by 228 · see citing works
Toward total quality of experience: A QoE model in a communication ecosystem
Laghari, K. U. R., & Connelly, K. H. (2012) · IEEE Communications Magazine · doi:10.1109/mcom.2012.6178834 · Cited by 223 · see citing works
CROWDMOS: An approach for crowdsourcing mean opinion score studies
Ribeiro, F. P., Florencio, D. A. F., Zhang, C., & Seltzer, M. L. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5946971 · Cited by 223 · see citing works
Standardization activities in the ITU for a QoE assessment of IPTV
Takahashi, A., Hands, D. S., & Barriac, V. (2008) · IEEE Communications Magazine · doi:10.1109/mcom.2008.4473087 · Cited by 219 · see citing works
A subjective study to evaluate video quality assessment algorithms
Seshadrinathan, K., Soundararajan, R., Bovik, A. C., et al. (2010) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.845382 · Cited by 213 · see citing works
Challenges of QoE management for cloud applications
Hobfeld, T., Schatz, R., Varela, M., & Timmerer, C. (2012) · IEEE Communications Magazine · doi:10.1109/mcom.2012.6178831 · Cited by 212 · see citing works
Factors Influencing Quality of Experience
Reiter, U., Brunnstrom, K. E., Moor, K. D., Larabi, M., Pereira, M., Pinheiro, A., You, J., & Žgank, A. (2014) · T-labs series in telecommunication services · doi:10.1007/978-3-319-02681-7_4 · Cited by 205 · see citing works
A crowdsourceable QoE evaluation framework for multimedia content
Chen, K.-T., Wu, C., Chang, Y.-C., & Lei, C. (2009) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1631272.1631339 · Cited by 192 · see citing works
HDR-VQM: An objective quality measure for high dynamic range video
Narwaria, M., Silva, M. P. D., & Callet, P. L. (2015) · Signal Processing Image Communication · doi:10.1016/j.image.2015.04.009 · Cited by 177 · see citing works
QoE in Video Transmission: A User Experience-Driven Strategy
Zhao, T., Liu, Q., & Chen, C. W. (2017) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2016.2619982 · Cited by 171 · see citing works
Egan, D., Brennan, S., Barrett, J., Qiao, Y., Timmerer, C., & Murray, N. (2016) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2016.7498964 · Cited by 170 · see citing works
QoE Prediction Model and its Application in Video Quality Adaptation Over UMTS Networks
Khan, A., Sun, L., & Ifeachor, E. (2011) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2011.2176324 · Cited by 169 · see citing works
Inferring the QoE of HTTP video streaming from user-viewing activities
Mok, R. K. P., Chan, E. W. W., Luo, X., & Chang, R. K. C. (2011) · doi:10.1145/2018602.2018611 · Cited by 163 · see citing works
Context-Aware QoE Modelling, Measurement, and Prediction in Mobile Computing Systems
Mitra, K., Zaslavsky, A., & Ahlund, C. (2015) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2013.155 · Cited by 144 · see citing works
Impact of frame rate and resolution on objective QoE metrics
Zinner, T., Hohlfeld, O., Abboud, O., & Hoßfeld, T. (2010) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2010.5518277 · Cited by 138 · see citing works
Quality of Experience Measurements for Video Streaming over Wireless Networks
Piamrat, K., Viho, C., Bonnin, J., & Ksentini, A. (2009) · International Conference on Information Technology: New Generations · doi:10.1109/itng.2009.121 · Cited by 132 · see citing works
Survey on Qoe\Qos Correlation Models Formultimedia Services
Alreshoodi, M. A., & Woods, J. (2013) · International Journal of Distributed and Parallel systems · doi:10.5121/ijdps.2013.4305 · Cited by 115 · see citing works
Display device-adapted video quality-of-experience assessment
Rehman, A., Zeng, K., & Wang, Z. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2077917 · Cited by 113 · see citing works
Quadrant of euphoria: a crowdsourcing platform for QoE assessment
Chen, K.-T., Chang, C.-J., Wu, C., et al. (2010) · IEEE Network · doi:10.1109/mnet.2010.5430141 · Cited by 112 · see citing works
An Overview of Quality of Experience Measurement Challenges for Video Applications in IP Networks
Serral-Gracià, R., Cerqueira, E., Curado, M., Yannuzzi, M., Monteiro, E., & Masip‐Bruin, X. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13315-2_21 · Cited by 107 · see citing works
Agboma, F., & Liotta, A. (2008) · International Conference on Advances in Mobile Computing and Multimedia (MoMM) · doi:10.1145/1497185.1497210 · Cited by 105 · see citing works
The memory effect and its implications on Web QoE modeling
Hosfeld, T., Biedermann, S., Schatz, R., Platzer, A., Egger, S., & Fiedler, M. (2011) · International Teletraffic Congress · doi:10.5555/2043468.2043485 · Cited by 104 · see citing works
Empirical study based on machine learning approach to assess the QoS/QoE correlation
Mushtaq, M. S., Augustin, B., & Mellouk, A. (2012) · 2012 17th European Conference on Networks and Optical Communications · doi:10.1109/noc.2012.6249939 · Cited by 100 · see citing works
Predicting quality of experience in multimedia streaming
Menkovski, V., Oredope, A., Liotta, A., & Sánchez, A. C. (2009) · International Conference on Advances in Mobile Computing and Multimedia (MoMM) · doi:10.1145/1821748.1821766 · Cited by 91 · see citing works
Towards subjective quality of experience assessment for omnidirectional video streaming
Schatz, R., Sackl, A., Timmerer, C., & Gardlo, B. (2017) · 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2017.7965657 · Cited by 88 · see citing works
A survey of QoE assurance in converged networks
Stankiewicz, R., & Jajszczyk, A. (2011) · Computer Networks · doi:10.1016/j.comnet.2011.02.004 · Cited by 83 · see citing works
Quality of experience management in mobile content delivery systems
Agboma, F., & Liotta, A. (2010) · Telecommunication Systems · doi:10.1007/s11235-010-9355-6 · Cited by 77 · see citing works
Review of objective video quality metrics and performance comparison using different databases
Vranješ, M., Rimac-Drlje, S., & Grgić, K. (2012) · Signal Processing Image Communication · doi:10.1016/j.image.2012.10.003 · Cited by 71 · see citing works
Quantifying the Quality of Audio and Video Transmissions over the Internet: The PSQA Approach
Rubino, G. (2006) · IMPERIAL COLLEGE PRESS eBooks · doi:10.1142/9781860948947_0014 · Cited by 62 · see citing works
QoS arbitration for improving the QoE in multimedia transmission
Siller, M. (2003) · doi:10.1049/cp:20030531 · Cited by 57 · see citing works
A multi-dimensional view of QoE: the ARCU model
Skorin‐Kapov, L., & Varela, M. (2012) · Cited by 55 · see citing works
Towards a comprehensive framework for QOE and user behavior modelling
Reichl, P., Egger, S., Möller, S., et al. (2015) · International Workshop on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2015.7148138 · Cited by 54 · see citing works
Volk, M., Sterle, J., Sedlar, U., & Kos, A. (2010) · IEEE Communications Magazine · doi:10.1109/mcom.2010.5534597 · Cited by 54 · see citing works
Survey of web-based crowdsourcing frameworks for subjective quality assessment
Hoßfeld, T., Hirth, M., Korshunov, P., et al. (2014) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2014.6958831 · Cited by 49 · see citing works
Testolin, A., Zanforlin, M., De Filippo De Grazia, M., Munaretto, D., Zanella, A., Zorzi, M., & Zorzi, M. (2014) · 2014 13th Annual Mediterranean Ad Hoc Networking Workshop (MED-HOC-NET) · doi:10.1109/medhocnet.2014.6849102 · Cited by 45 · see citing works
Controlling Multimedia QoS in the Future Home Network Using the PSQA Metric
Rubino, G., Varela, M., & Bonnin, J.-M. (2006) · The Computer Journal · doi:10.1093/comjnl/bxh165 · Cited by 40 · see citing works
Accurate and efficient SLA compliance monitoring
Sommers, J., Barford, P., Duffield, N. G., et al. (2007) · ACM SIGCOMM Computer Communication Review · doi:10.1145/1282427.1282394 · Cited by 36 · see citing works
A_PSQA: Efficient real-time video streaming QoE tool in a future media internet context
Chérif, W., Ksentini, A., Négru, D., & Sidibé, M. (2011) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2011.6011993 · Cited by 34 · see citing works
The effectiveness of a QoE-based video output scheme for audio-video ip transmission
Tasaka, S., Yoshimi, H., Hirashima, A., et al. (2008) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1459359.1459395 · Cited by 33 · see citing works
Modelling Human Factors in Perceptual Multimedia Quality
Scott, M., Guntuku, S. C., Yang, H., Lin, W., & Ghinea, G. (2015) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2733373.2806254 · Cited by 32 · see citing works
Ghareeb, M., & Viho, C. (2010) · Annual Communication Networks and Services Research Conference (CNSR) · doi:10.1109/cnsr.2010.15 · Cited by 30 · see citing works
Moor, K. D., Joseph, W., Ketykó, I., Tanghe, E., Deryckere, T., Martens, L., & Marez, L. D. (2010) · EURASIP Journal on Wireless Communications and Networking · doi:10.1155/2010/541568 · Cited by 26 · see citing works
A framework of scalable QoE modeling for application explosion in the Internet of Things
Ikeda, Y., Kouno, S., Shiozu, A., & Noritake, K. (2016) · 2016 IEEE 3rd World Forum on Internet of Things (WF-IoT) · doi:10.1109/wf-iot.2016.7845393 · Cited by 25 · see citing works
Impact of screening technique on crowdsourcing QoE assessments
Gardlo, B., Ries, M., & Hoßfeld, T. (2012) · International Conference Radioelektronika · Cited by 24 · see citing works
Han, B., Zhang, X., Qi, Y., Gao, Y., & Yang, D. (2012) · Wireless Personal Communications · doi:10.1007/s11277-012-0739-7 · Cited by 24 · see citing works
Understanding User Experience of Mobile Video: Framework, Measurement, and Optimization
Song, W., Tjondronegoro, D., & Docherty, M. (2012) · InTech eBooks · doi:10.5772/39053 · Cited by 23 · see citing works
Sedano, I., Brunnstrom, K. E., Kihl, M., et al. (2014) · EURASIP Journal on Image and Video Processing · doi:10.1186/1687-5281-2014-4 · Cited by 21 · see citing works
A_PSQA: PESQ-like non-intrusive tool for QoE prediction in VoIP services
Chérif, W., Ksentini, A., Négru, D., & Sidibé, M. (2012) · International Conference on Communications · doi:10.1109/icc.2012.6364004 · Cited by 20 · see citing works
Wang, M., Zhang, R., & Shen, X. (2015) · Wireless networks · doi:10.1007/978-3-319-25130-1 · Cited by 19 · see citing works
QoE Estimation for Web Service Selection Using a Fuzzy-Rough Hybrid Expert System
Pokhrel, J., Lalanne, F., Cavalli, A. R., et al. (2014) · IEEE International Conference on Advanced Information Networking and Applications (AINA) · doi:10.1109/aina.2014.77 · Cited by 18 · see citing works
Modeling immersive media experiences by sensing impact on subjects
Kroupi, E., Hanhart, P., Lee, J., Řeřábek, M., & Ebrahimi, T. (2015) · Multimedia Tools and Applications · doi:10.1007/s11042-015-2980-z · Cited by 18 · see citing works
QoE evaluation of high-definition IPTV services
Ries, M., Fröehlich, P., & Schatz, R. (2011) · doi:10.1109/radioelek.2011.5936485 · Cited by 15 · see citing works
A QoE Based Evaluation of Service Quality in Wireless Communication Network
Du, Y., Zhou, W., Chen, B., & Song, J. (2009) · International Conference on New Trends in Information Science and Service Science · doi:10.1109/niss.2009.163 · Cited by 14 · see citing works
QUVE: QoE Maximizing Framework for Video-Streaming
Kimura, T., Yokota, M., Matsumoto, A., et al. (2016) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2016.2632060 · Cited by 14 · see citing works
Modeling QoE in Dependable Tele-Immersive Applications: A Case Study of World Opera
Veeraragavan, N. R., Montecchi, L., Nostro, N., et al. (2015) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2015.2503291 · Cited by 12 · see citing works
State-of-the-Practice in Evaluation of Quality of Experience in Real-Life Environments
Matulin, M., & Mrvelj, Š. (2013) · PROMET - Traffic&Transportation · doi:10.7307/ptt.v25i3.1195 · Cited by 11 · see citing works
A new three-layer QoE modeling method for HTTP video streaming over wireless networks
Kai-yu, W., Yumei, W., & Zhang, L. (2014) · IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC) · doi:10.1109/icnidc.2014.7000265 · Cited by 11 · see citing works
An Ontology for the Quality of Experience framework
Gallo, E., Siller, M., & Woods, J. (2007) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2007.4414109 · Cited by 11 · see citing works
Flicker sensitive motion tuned video quality assessment
Choi, L. K., & Bovik, A. C. (2016) · IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI) · doi:10.1109/ssiai.2016.7459167 · Cited by 11 · see citing works
Event triggered intelligent video recording system using MS-SSIM for smart home security
Khalaf, H. A., Tolba, A. S., & Rashid, M. Z. (2016) · Ain Shams Engineering Journal · doi:10.1016/j.asej.2016.10.001 · Cited by 11 · see citing works
Examining subjective evaluation methods used in multimedia Quality of Experience research
Kunze, K., & Strohmeier, D. (2012) · International Workshop on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2012.6263838 · Cited by 6 · see citing works
Theoretical modelings for mobile web service QoE assessment
Zhang, M., Shen, Z., Zhang, X., Luan, L., & Ouyang, Q. (2015) · IEEE Wireless Communications and Networking Conference · doi:10.1109/wcnc.2015.7127779 · Cited by 4 · see citing works
Rule-based combination of video quality metrics
Gutiérrez, J., Raake, A., Argyropoulos, S., et al. (2015) · International Workshop on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2015.7148133 · Cited by 2 · see citing works
A multi-level QoE framework for smartphone video streaming applications
Chen, Y., Chang, J., & Wei, H. (2014) · IEEE Globecom Workshops (GC Wkshps) · doi:10.1109/glocomw.2014.7063435 · Cited by 2 · see citing works
Mobile video perception assessment model based on QoE
Yu, Q., & Sun, S. (2016) · International Symposium on Communications and Information Technologies · doi:10.1109/iscit.2016.7751712 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
QoE in IoT: a vision, survey and future directions
Fizza, K., Banerjee, A., Mitra, K., Jayaraman, P. P., Ranjan, R., Patel, P., & Georgakopoulos, D. (2021) · Discover Internet of Things · doi:10.1007/s43926-021-00006-7 · Cited by 107 · see citing works
- Authors
- Yang, M., Wang, S., Calheiros, R. N., & Yang, F.
- Affiliations
- Beijing University of Posts and Telecommunications (CN)
- Western Sydney University (AU)
- Venue
- IEEE Access
- Published
- 2018
- Volume
- 6
- Pages
- 48374-48390
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2889485247
- Cited by
- 54
- References
- 82
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 4 · 2023: 8 · 2022: 13 · 2021: 7 · 2020: 8 · 2019: 2 · 2018: 2
- FWCI
- 2.38
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Natural Science Foundation of China (61472047)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Traffic and Congestion Control
- Social Sciences › Sociology and Political Science › Multimedia Communication and Technology
- Keywords
- quality of experience, network services, subjective evaluation, objective assessment, hybrid evaluation, all-IP networks, multimedia services, user satisfaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
