On User-Centric Modular QoE Prediction for VoIP Based on Machine-Learning Algorithms
Abstract
The impact of the network performance on the quality of experience (QoE) for various services is not well-understood. Assessing the impact of different network and channel conditions on the user experience is important for improving the telecommunication services. The QoE for various wireless services including VoIP, video streaming, and web browsing, has been in the epicenter of recent networking activities. The majority of such efforts aim to characterize the user experience, analyzing various types of measurements often in an aggregate manner. This paper proposes the MLQoE, a modular algorithm for user-centric QoE prediction. The MLQoE employs multiple machine learning (ML) algorithms, namely, Artificial Neural Networks, Support Vector Regression machines, Decision Trees, and Gaussian Naive Bayes classifiers, and tunes their hyper-parameters. It uses the Nested Cross Validation (nested CV) protocol for selecting the best classifier and the corresponding best hyper-parameter values and estimates the performance of the final model. The MLQoE is conservative in the performance estimation despite multiple induction of models. The MLQoE is modular, in that, it can be easily extended to include other ML algorithms. The MLQoE selects the ML algorithm that exhibits the best performance and its parameters automatically given the dataset used as input. It uses empirical measurements based on network metrics (e.g., packet loss, delay, and packet interarrival) and subjective opinion scores reported by actual users. This paper extensively evaluates the MLQoE using three unidirectional datasets containing VoIP calls over wireless networks under various network conditions and feedback from subjects (collected in field studies). Moreover, it performs a preliminary analysis to assess the generality of our methodology using bidirectional VoIP and video traces. The MLQoE outperforms several state-of-the-art algorithms, resulting in fairly accurate predictions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In terms of mean absolute error, supervised learning models generally outperform the E-model and even the full-reference PESQ, with only one exception observed with dataset 2.
“Experiments show that, in terms of mean absolute error MAE, the supervised learning models generally outperform E-model and even the full-reference PESQ, only one exception is observed with dataset 2.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [89]
According to [89], RF is a better classifier than NN when it comes to predicting MOS.
“According to [89], RF is a better classifier than NN when it comes to predicting MOS.”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [89]
Some works find it sufficient to correlate QoE to a single network QoS parameter, while others argue that multiple QoS parameters impact QoE and need to be considered in tandem.
“While some find it sufficient to correlate the QoE to a single network QoS parameter [145, 240, 383, 408], e.g.delay or throughput, others argue that multiple QoS parameters impact the QoE and need to be considered in tandem as features in a QoE/QoS correlation model [89,102,114,235,287,328,432].”
From A comprehensive survey on machine learning for networking: Evolution, applications and research opportunities (2018) · cites this paper as [89]
Cites
(1967) · Cited by 23,719 · see citing works
A tutorial on support vector regression
Smola, A. J., & Schölkopf, B. (2004) · Statistics and Computing · doi:10.1023/b:stco.0000035301.49549.88 · Cited by 13,482 · see citing works
Finding groups in data: an introduction to cluster analysis
(1991) · Choice Reviews Online · doi:10.5860/choice.28-4558 · Cited by 10,537 · see citing works
Artificial neural networks: a tutorial
Jain, A. K., Mao, J., & Mohiuddin, K. M. (1996) · Computer · doi:10.1109/2.485891 · Cited by 4,989 · see citing works
Causation, Prediction, and Search
Spirtes, P., Glymour, C., & Scheines, R. (1993) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-2748-9 · Cited by 4,617 · see citing works
Causation, Prediction, and Search
Spirtes, P., Glymour, C., & Scheines, R. (2001) · doi:10.7551/mitpress/1754.001.0001 · Cited by 4,521 · see citing works
Image information and visual quality
Sheikh, H. R., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.859378 · Cited by 4,104 · see citing works
Pattern Classification (2nd ed.)
Duda, R. O., Hart, P. E., & Stork, D. G. (1999) · Cited by 2,509 · see citing works
Barker, L. E. (2002) · Technometrics · doi:10.1198/tech.2002.s659 · Cited by 2,482 · see citing works
Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses
Niknian, M., & Good, P. (1995) · Technometrics · doi:10.2307/1269919 · Cited by 1,406 · see citing works
A new standardized method for objectively measuring video quality
Pinson, M. H., & Wolf, S. (2004) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2004.834028 · Cited by 1,182 · cited 1× · see citing works
“Typically, studies that perform subjective tests estimate the performance of their models using the simple hold-out estimation [20], [21] or the cross-validation algorithm [22], [23], [24].”
BackgroundStatnikov, A., Aliferis, C., Tsamardinos, I., Hardin, D. P., & Levy, S. (2004) · Bioinformatics · doi:10.1093/bioinformatics/bti033 · Cited by 837 · see citing works
Self-citationMotion Tuned Spatio-Temporal Quality Assessment of Natural Videos
Seshadrinathan, K., & Bovik, A. C. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2034992 · Cited by 708 · see citing works
Understanding the impact of video quality on user engagement
Dobrian, F., Sekar, V., Awan, A., Stoica, I. G., Joseph, D. A., Ganjam, A., Zhan, J., & Zhang, H. (2011) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2018436.2018478 · Cited by 674 · see citing works
Using psychophysiological techniques to measure user experience with entertainment technologies
Mandryk, R. L., Inkpen, K. M., & Calvert, T. W. (2006) · Behaviour and Information Technology · doi:10.1080/01449290500331156 · Cited by 516 · see citing works
SyncScan: practical fast handoff for 802.11 infrastructure networks
Ramani, I., & Savage, S. (2005) · doi:10.1109/infcom.2005.1497933 · Cited by 494 · see citing works
Aliferis, C., Statnikov, A., Tsamardinos, I., Mani, S., & Koutsoukos, X. (2010) · doi:10.5555/1756006.1756013 · Cited by 482 · see citing works
Self-citationFinding Groups in Data: An Introduction to Cluster Analysis.
Fox, W. R., Kaufman, L., & Rousseeuw, P. J. (1991) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.2307/2347530 · Cited by 452 · see citing works
Video Stream Quality Impacts Viewer Behavior: Inferring Causality Using Quasi-Experimental Designs
Krishnan, S. S., & Sitaraman, R. K. (2013) · IEEE/ACM Transactions on Networking · doi:10.1109/tnet.2013.2281542 · Cited by 429 · see citing works
Permutation Tests: A Practical Guide to Resampling Methods for Testing Hypotheses.
Modarres, R., & Good, P. (1995) · Journal of the American Statistical Association · doi:10.2307/2291167 · Cited by 347 · see citing works
Understanding the impact of video quality on user engagement
Dobrian, F., Sekar, V., Awan, A., Stoica, I. G., Joseph, D. A., Ganjam, A., Zhan, J., & Zhang, H. (2011) · ACM SIGCOMM Computer Communication Review · doi:10.1145/2043164.2018478 · Cited by 332 · see citing works
Theoretical maximum throughput of IEEE 802.11 and its applications
Jun, J., Peddabachagari, P., & Sichitiu, M. L. (2003) · doi:10.1109/nca.2003.1201163 · Cited by 326 · see citing works
Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies
Moorthy, A. K., Choi, L. K., Bovik, A. C., & Veciana, G. A. D. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2212417 · Cited by 304 · see citing works
The Logarithmic Nature of QoE and the Role of the Weber-Fechner Law in QoE Assessment
Reichl, P., Egger, S., Schatz, R., & D'Alconzo, A. (2010) · 2010 IEEE International Conference on Communications · doi:10.1109/icc.2010.5501894 · Cited by 284 · see citing works
Piuri, V., & Alippi, C. (1998) · Journal of Systems Architecture · doi:10.1016/s1383-7621(97)00063-5 · Cited by 234 · see citing works
Statnikov, A., Tsamardinos, I., Dosbayev, Y., & Aliferis, C. (2005) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2005.05.002 · Cited by 199 · see citing works
Self-citationReichl, P., Tuffin, B., & Schatz, R. (2011) · Telecommunication Systems · doi:10.1007/s11235-011-9503-7 · Cited by 182 · see citing works
Quality of experience in distributed interactive multimedia environments
Wu, W., Arefin, A., Rivas, R., Nahrstedt, K., Sheppard, R. M., & Yang, Z. (2009) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1631272.1631338 · Cited by 178 · see citing works
Quality of Experience: Advanced Concepts, Applications and Methods
(2014) · T-Labs Series in Telecommunication Services · doi:10.1007/978-3-319-02681-7 · Cited by 155 · see citing works
Voice quality prediction models and their application in VoIP networks
Sun, L., & Ifeachor, E. (2006) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.876279 · Cited by 155 · see citing works
Context-Aware QoE Modelling, Measurement, and Prediction in Mobile Computing Systems
Mitra, K., Zaslavsky, A., & Ahlund, C. (2015) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2013.155 · Cited by 144 · see citing works
OneClick: A Framework for Measuring Network Quality of Experience
Chen, K.-T., Tu, C.-C., & Xiao, W. C. (2009) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infcom.2009.5061978 · Cited by 127 · see citing works
Aggarwal, V., Halepovic, E., Pang, J., Venkataraman, S., & Yan, H. (2014) · IEEE Workshop on Mobile Computing Systems and Applications (WMCSA) · doi:10.1145/2565585.2565600 · Cited by 124 · see citing works
Tsamardinos, I., Rakhshani, A., & Lagani, V. (2015) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213015400230 · Cited by 111 · see citing works
Self-citationThe Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study
Pinson, M., Janowski, L., Pépion, R., Huynh‐Thu, Q., Schmidmer, C., Corriveau, P., Younkin, A. C., Callet, P. L., Barkowsky, M., & Ingram, W. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2215306 · Cited by 96 · see citing works
Predicting quality of experience in multimedia streaming
Menkovski, V., Oredope, A., Liotta, A., & Sánchez, A. C. (2009) · International Conference on Advances in Mobile Computing and Multimedia (MoMM) · doi:10.1145/1821748.1821766 · Cited by 91 · see citing works
Does context matter in quality evaluation of mobile television?
Pyykko, S. J., & Hannuksela, M. M. (2008) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/1409240.1409248 · Cited by 80 · see citing works
Sun, L., & Ifeachor, E. (2004) · International Conference on Communications · doi:10.1109/icc.2004.1312757 · Cited by 80 · see citing works
Do Users Always Know What’s Good For Them? Utilising Physiological Responses to Assess Media Quality
Wilson, G. M., & Sasse, M. A. (2000) · Springer eBooks · doi:10.1007/978-1-4471-0515-2_22 · Cited by 79 · see citing works
Hoßfeld, T., Hock, D., Tran‐Gia, P., Tutschku, K., & Fiedler, M. (2008) · Cited by 70 · see citing works
Perceived speech quality prediction for voice over IP-based networks
Sun, L., & Ifeachor, E. (2003) · doi:10.1109/icc.2002.997307 · Cited by 66 · see citing works
Experimental Measurement of the Capacity for VoIP Traffic in IEEE 802.11 WLANs
Shin, S., & Schulzrinne, H. G. (2007) · doi:10.1109/infcom.2007.234 · Cited by 51 · see citing works
Menkovski, V., Exarchakos, G., & Liotta, A. (2010) · International Workshop on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2010.5517692 · Cited by 50 · see citing works
Objective Assessment Methodology for Estimating Conversational Quality in VoIP
Takahashi, A., 倉島, 敦., & Yoshino, H. (2006) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2006.883261 · Cited by 49 · see citing works
An empirical evaluation of VoIP playout buffer dimensioning in Skype, Google talk, and MSN Messenger
Wu, C., Chen, K.-T., Huang, C., et al. (2009) · International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV) · doi:10.1145/1542245.1542268 · Cited by 47 · see citing works
Evaluating Users’ Satisfaction in Packet Networks Using Random Neural Networks
Rubino, G., Tirilly, P., & Varela, M. (2006) · Lecture notes in computer science · doi:10.1007/11840817_32 · Cited by 45 · see citing works
Quality of experience evaluation of voice communication: an affect-based approach
Bhattacharya, A., Wu, W., & Yang, Z. (2012) · Human-Centric Computing and Information Sciences · doi:10.1186/2192-1962-2-7 · Cited by 44 · see citing works
Tsamardinos, I., Rakhshani, A., & Lagani, V. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07064-3_1 · Cited by 42 · see citing works
Self-citationPredicting user dissatisfaction with Internet application performance at end-hosts
Joumblatt, D., Chandrashekar, J., Kveton, B., et al. (2013) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infcom.2013.6566770 · Cited by 39 · see citing works
Structure-based variable selection for survival data
Lagani, V., & Tsamardinos, I. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq261 · Cited by 29 · see citing works
Self-citationOptimized online learning for QoE prediction
Menkovski, V., Oredope, A., Liotta, A., et al. (2009) · TU/e Research Portal · Cited by 24 · see citing works
A_PSQA: PESQ-like non-intrusive tool for QoE prediction in VoIP services
Chérif, W., Ksentini, A., Négru, D., & Sidibé, M. (2012) · International Conference on Communications · doi:10.1109/icc.2012.6364004 · Cited by 20 · see citing works
Supporting wireless access markets with a user-centric QoE-based geo-database
Fortetsanakis, G., Katsarakis, M., Plakia, M., et al. (2012) · doi:10.1145/2348676.2348685 · Cited by 11 · see citing works
Self-citationQoE estimation and prediction using hidden Markov models in heterogeneous access networks
Mitra, K., Åhlund, C., & Zaslavsky, A. (2012) · Australasian Telecommunication Networks and Applications Conference · doi:10.1109/atnac.2012.6398083 · Cited by 10 · see citing works
A Comparative Analysis of the Perceived Quality of VoIP under Various Wireless Network Conditions
Tsompanidis, I., Fortetsanakis, G., Hirvonen, T., et al. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13315-2_28 · Cited by 9 · see citing works
Analyzing the impact of various wireless network conditions on the perceived quality of VoIP
Tsompanidis, I., Fortetsanakis, G., Hirvonen, T., et al. (2010) · IEEE Workshop on Local and Metropolitan Area Networks (LANMAN) · doi:10.1109/lanman.2010.5507146 · Cited by 7 · see citing works
Hecht, J. (2014) · IEEE Spectrum · doi:10.1109/mspec.2014.6905488 · Cited by 7 · see citing works
Evaluation of a User-centric QoE-based Recommendation Tool for Wireless Access
Katsarakis, M., Theodosiadis, V., & Papadopouli, M. (2015) · doi:10.1145/2787394.2787402 · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Boutaba, R., Salahuddin, M. A., Limam, N., Ayoubi, S., Shahriar, N., Estrada-Solano, F., & Caicedo, O. M. (2018) · Journal of Internet Services and Applications · doi:10.1186/s13174-018-0087-2 · Cited by 1,036 · see citing works
- Authors
- Charonyktakis, P., Plakia, M., Tsamardinos, I., & Papadopouli, M.
- Affiliations
- University of Crete (GR)
- Venue
- IEEE Transactions on Mobile Computing
- Published
- 2016-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1443-1456
- ISSN
- 1536-1233 (ISSN-L)
- OpenAlex
- W2351381307
- Cited by
- 75
- References
- 58
- Citations by year
- 2026: 1 · 2024: 2 · 2023: 9 · 2022: 8 · 2021: 7 · 2020: 9 · 2019: 12 · 2018: 14 · 2017: 10 · 2016: 3
- FWCI
- 7.81
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Mobile Computing (publishedVersion)
- Funders
- General Secretariat for Research and Technology
- Research Excellence Investigator-driven Grant
- ERC
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Traffic and Congestion Control
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Advanced Wireless Network Optimization
- Keywords
- quality of experience prediction, VoIP, wireless networks, network metrics, packet loss, delay, classifier selection, hyperparameter optimization, nested cross-validation, artificial neural networks, support vector regression, decision tree, Gaussian naive Bayes, video traces
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
