Recurrent and Dynamic Models for Predicting Streaming Video Quality of Experience
Abstract
Streaming video services represent a very large fraction of global bandwidth consumption. Due to the exploding demands of mobile video streaming services, coupled with limited bandwidth availability, video streams are often transmitted through unreliable, low-bandwidth networks. This unavoidably leads to two types of major streaming-related impairments: compression artifacts and/or rebuffering events. In streaming video applications, the end-user is a human observer; hence being able to predict the subjective Quality of Experience (QoE) associated with streamed videos could lead to the creation of perceptually optimized resource allocation strategies driving higher quality video streaming services. We propose a variety of recurrent dynamic neural networks that conduct continuous-time subjective QoE prediction. By formulating the problem as one of time-series forecasting, we train a variety of recurrent neural networks and non-linear autoregressive models to predict QoE using several recently developed subjective QoE databases. These models combine multiple, diverse neural network inputs, such as predicted video quality scores, rebuffering measurements, and data related to memory and its effects on human behavioral responses, using them to predict QoE on video streams impaired by both compression artifacts and rebuffering events. Instead of finding a single time-series prediction model, we propose and evaluate ways of aggregating different models into a forecasting ensemble that delivers improved results with reduced forecasting variance. We also deploy appropriate new evaluation metrics for comparing time-series predictions in streaming applications. Our experimental results demonstrate improved prediction performance that approaches human performance. An implementation of this work can be found at https://github.com/christosbampis/NARX_QoE_release.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · see citing works
Self-citationNonparametric Statistics for the Behavioral Sciences
Guitton, H., & Siegel, S. (1958) · Revue économique · doi:10.2307/3498751 · Cited by 35,281 · see citing works
An Algorithm for Least-Squares Estimation of Nonlinear Parameters
Marquardt, D. W. (1963) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0111030 · Cited by 30,728 · see citing works
Book Review: Time Series Analysis: Forecasting and Control
Geurts, M. D., Box, G. E. P., & Jenkins, G. M. (1977) · Journal of Marketing Research · doi:10.2307/3150485 · Cited by 19,399 · see citing works
A method for the solution of certain non-linear problems in least squares
Levenberg, K. (1944) · Quarterly of Applied Mathematics · doi:10.1090/qam/10666 · Cited by 12,367 · see citing works
Elman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · see citing works
Identification and control of dynamical systems using neural networks
Narendra, K. S., & Parthasarathy, K. (1990) · IEEE Transactions on Neural Networks · doi:10.1109/72.80202 · Cited by 8,031 · see citing works
Nonparametric statistics for the behavioral sciences
McGinn, L. C. (1957) · Journal of the Franklin Institute · doi:10.1016/0016-0032(57)90766-4 · Cited by 7,284 · see citing works
Making a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766 · see citing works
Self-citationMultiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253 · cited 1× · see citing works
“1) [32], MS-SSIM [76], SSIM [77] and ST-RRED [33] (see Table V).”
Self-citationTime series forecasting using a hybrid ARIMA and neural network model
Zhang, G. P. (2003) · Neurocomputing · doi:10.1016/s0925-2312(01)00702-0 · Cited by 4,576 · see citing works
(1994) · Knowledge Discovery and Data Mining · Cited by 3,137 · see citing works
Long short-term memory recurrent neural network architectures for large scale acoustic modeling
Sak, H., Senior, A., & Beaufays, F. (2014) · Interspeech 2014 · doi:10.21437/interspeech.2014-80 · Cited by 2,973 · see citing works
A Novel Connectionist System for Unconstrained Handwriting Recognition
Graves, A., Liwicki, M., Fernandez, S., Bertolami, R., Horst Bunke, A., & Schmidhuber, J. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.137 · Cited by 1,984 · see citing works
Ensembling neural networks: Many could be better than all
Zhou, Z.-H., Wu, J., & Tang, W. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(02)00190-x · Cited by 1,964 · see citing works
Neural Network Ensembles, Cross Validation, and Active Learning
Krogh, A., & Vedelsby, J. (1994) · Cited by 1,838 · see citing works
Neural Adaptive Video Streaming with Pensieve
Mao, H., Netravali, R., & Alizadeh, M. (2017) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3098822.3098843 · Cited by 1,378 · see citing works
Study of Subjective and Objective Quality Assessment of Video
Seshadrinathan, K., Soundararajan, R., Bovik, A. C., & Cormack, L. K. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2042111 · Cited by 1,160 · see citing works
Self-citationCombination forecasts of output growth in a seven‐country data set
Stock, J. H., & Watson, M. W. (2004) · Journal of Forecasting · doi:10.1002/for.928 · Cited by 1,139 · see citing works
Video quality assessment based on structural distortion measurement
Wang, Z., Lu, L., & Bovik, A. C. (2004) · Signal Processing: Image Communication · doi:10.1016/s0923-5965(03)00076-6 · Cited by 1,024 · see citing works
Self-citationA buffer-based approach to rate adaptation
Huang, T.-Y., Ramesh Johari, B., McKeown, N., Trunnell, M., & Watson, M. (2014) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/2619239.2626296 · Cited by 1,007 · see citing works
Leutbecher, M., & Palmer, T. N. (2007) · Journal of Computational Physics · doi:10.1016/j.jcp.2007.02.014 · Cited by 920 · see citing works
A Survey on Quality of Experience of HTTP Adaptive Streaming
Seufert, M., Egger, S., Slanina, M., Zinner, T., Hoßfeld, T., & Tran-Gia, P. (2015) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2014.2360940 · Cited by 844 · see citing works
Learning long-term dependencies in NARX recurrent neural networks
Lin, T.-N., Horne, B. G., Peter Tiňo, B., & Giles, C. L. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.548162 · Cited by 788 · see citing works
Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos
Seshadrinathan, K., & Bovik, A. C. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2034992 · Cited by 708 · cited 1× · see citing works
“g, in [27] and in hybrid models of retrospective QoE [45], [46],”
BackgroundSelf-citationProbe and Adapt: Rate Adaptation for HTTP Video Streaming At Scale
Li, Z., Zhu, X., Gahm, J., Pan, R., Hu, H., Begen, A. C., & Oran, D. (2014) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2014.140405 · Cited by 613 · see citing works
Self-citationBlind Prediction of Natural Video Quality
Saad, M. A., Bovik, A. C., & Charrier, C. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2299154 · Cited by 471 · see citing works
Self-citationComputational capabilities of recurrent NARX neural networks
Siegelmann, H. T., Horne, B. G., & Giles, C. L. (1997) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/3477.558801 · Cited by 458 · see citing works
Video Quality Assessment by Reduced Reference Spatio-Temporal Entropic Differencing
Soundararajan, R., & Bovik, A. C. (2013) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2012.2214933 · Cited by 368 · cited 2× · see citing works
“1) [32], MS-SSIM [76], SSIM [77] and ST-RRED [33] (see Table V).”
BackgroundSelf-citationA Completely Blind Video Integrity Oracle
Mittal, A., Saad, M. A., & Bovik, A. C. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2502725 · Cited by 357 · see citing works
Self-citationQuantification of YouTube QoE via Crowdsourcing
Hoßfeld, T., Seufert, M., Hirth, M., Zinner, T., Tran‐Gia, P., & Schatz, R. (2011) · IEEE International Symposium on Multimedia (ISM) · doi:10.1109/ism.2011.87 · Cited by 338 · see citing works
Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies
Moorthy, A. K., Choi, L. K., Bovik, A. C., & Veciana, G. A. D. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2212417 · Cited by 304 · see citing works
Self-citationAnalysis of Public Image and Video Databases for Quality Assessment
Winkler, S. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2215007 · Cited by 298 · see citing works
Neural network ensemble operators for time series forecasting
Kourentzes, N., Barrow, D. K., & Crone, S. F. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.12.011 · Cited by 294 · see citing works
Quality of experience for HTTP adaptive streaming services
Oyman, Ö., & Singh, S. (2012) · IEEE Communications Magazine · doi:10.1109/mcom.2012.6178830 · Cited by 284 · see citing works
Initial delay vs. interruptions: Between the devil and the deep blue sea
Hoßfeld, T., Egger, S., Schatz, R., Fiedler, M., Masuch, K., & Lorentzen, C. (2012) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2012.6263849 · Cited by 243 · see citing works
Automatic Prediction of Perceptual Image and Video Quality
Bovik, A. C. (2013) · Proceedings of the IEEE · doi:10.1109/jproc.2013.2257632 · Cited by 212 · see citing works
Self-citationTime Series Analysis Forecasting and Control
Rao, J. N. K., Box, G. E. P., & Jenkins, G. M. (1972) · Econometrica · doi:10.2307/1912100 · Cited by 199 · see citing works
Study of Temporal Effects on Subjective Video Quality of Experience
Bampis, C. G., Li, Z., Moorthy, A. K., Katsavounidis, I. X., Aaron, A., & Bovik, A. C. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2729891 · Cited by 188 · see citing works
Self-citationA Quality-of-Experience Index for Streaming Video
Duanmu, Z., Zeng, K., Ma, K., Rehman, A., & Wang, Z. (2016) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2016.2608329 · Cited by 185 · see citing works
A spatiotemporal most-apparent-distortion model for video quality assessment
Vu, P. V., Vu, C. T., & Chandler, D. M. (2011) · 2011 18th IEEE International Conference on Image Processing · doi:10.1109/icip.2011.6116171 · Cited by 180 · see citing works
SpEED-QA: Spatial Efficient Entropic Differencing for Image and Video Quality
Bampis, C. G., Gupta, P., Soundararajan, R., & Bovik, A. C. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2726542 · Cited by 150 · see citing works
Self-citationQuality of experience estimation for adaptive HTTP/TCP video streaming using H.264/AVC
Singh, K., Hadjadj‐Aoul, Y., & Rubino, G. (2012) · IEEE Consumer Communications and Networking Conference · doi:10.1109/ccnc.2012.6181070 · Cited by 132 · see citing works
A bitstream-based, scalable video-quality model for HTTP adaptive streaming: ITU-T P.1203.1
Raake, A., Garcia, M.-N., Robitza, W., et al. (2017) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2017.7965631 · Cited by 124 · see citing works
Quality of experience and HTTP adaptive streaming: A review of subjective studies
Garcia, M. N., Francesca, D. S., Tavakoli, S., Staelens, N., Egger, S., Brunnstrom, K. E., & Raake, A. (2014) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2014.6982310 · Cited by 96 · see citing works
Modeling the Time—Varying Subjective Quality of HTTP Video Streams With Rate Adaptations
Chen, C., Choi, L. K., de Veciana, G., Caramanis, C., Heath, R. W., & Bovik, A. C. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2312613 · Cited by 94 · see citing works
Self-citationAn Optical Flow-Based Full Reference Video Quality Assessment Algorithm
Manasa, K., & Channappayya, S. S. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2548247 · Cited by 93 · see citing works
Sporadic frame dropping impact on quality perception
Pastrana-Vidal, R. R., Gicquel, J. C., Colomes, C., et al. (2004) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.525746 · Cited by 89 · see citing works
Recency and duration neglect in subjective assessment of television picture quality
Hands, D. S., & Avons, S. E. (2001) · Applied Cognitive Psychology · doi:10.1002/acp.731 · Cited by 85 · see citing works
Quality metric to assess video streaming service over TCP considering temporal location of pauses
Rodríguez, D. Z., Abrahão, J. I., Begazo, D. C., et al. (2012) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2012.6311346 · Cited by 71 · see citing works
Modeling the Time—Varying Subjective Quality of HTTP Video Streams With Rate Adaptations
Chen, C., Member, S., Choi, L. K., Member, S., Veciana, G. A. D., Caramanis, C., Heath, R. W., & Bovik, A. C. (2014) · Cited by 69 · see citing works
Self-citationStudy of the effects of stalling events on the quality of experience of mobile streaming videos
Ghadiyaram, D., Bovik, A. C., Yeganeh, H., et al. (2014) · IEEE Global Conference on Signal and Information Processing · doi:10.1109/globalsip.2014.7032269 · Cited by 66 · see citing works
Self-citationA fusion-based video quality assessment (fvqa) index
Lin, J. Y., Liu, T.-J., Wu, E. C.-H., & Kuo, C. ‐. J. (2014) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipa.2014.7041705 · Cited by 65 · see citing works
A novel objective no-reference metric for digital video quality assessment
Yang, F., Wan, S., Chang, Y., et al. (2005) · IEEE Signal Processing Letters · doi:10.1109/lsp.2005.855553 · Cited by 64 · see citing works
Temporal Video Quality Model Accounting for Variable Frame Delay Distortions
Pinson, M., Choi, L. K., & Bovik, A. C. (2014) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2014.2365260 · Cited by 63 · see citing works
Self-citationContinuous Prediction of Streaming Video QoE Using Dynamic Networks
Bampis, C. G., Li, Z., & Bovik, A. C. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2705423 · Cited by 61 · see citing works
Self-citationStaelens, N., Meulenaere, J. D., Claeys, M., Wallendael, G. V., den Broeck, W. V., Cock, J. D., Rik Van De Walle, B., Demeester, P. M. A., & Turck, F. D. (2014) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2014.2359255 · Cited by 59 · see citing works
Bęben, A., Wiśniewski, P., Batalla, J. M., & Krawiec, P. (2016) · ACM Multimedia Systems Conference (MMSys) · doi:10.1145/2910017.2910596 · Cited by 55 · see citing works
Tavakoli, S., Egger, S., Seufert, M., et al. (2016) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2016.2577361 · Cited by 52 · see citing works
Learning to Predict Streaming Video QoE: Distortions, Rebuffering and MemoryPreprint
Bampis, C. G., & Bovik, A. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00633 · Cited by 52 · see citing works
Self-citationRecent developments in visual quality monitoring by key performance indicators
Leszczuk, M., Hanusiak, M., Farias, M. C. Q., Wyckens, E., & Heston, G. (2014) · Multimedia Tools and Applications · doi:10.1007/s11042-014-2229-2 · Cited by 43 · see citing works
Estimating the impact of single and multiple freezes on video quality
van Kester, S., Xiao, T., Kooij, R. E., Brunnstrom, K. E., & Ahmed, O. K. (2011) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.873390 · Cited by 36 · see citing works
A time-varying subjective quality model for mobile streaming videos with stalling events
Ghadiyaram, D., Pan, J., & Bovik, A. C. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2188882 · Cited by 35 · see citing works
Self-citationPrimacy Versus Recency in a Quantitative Model: Activity Is the Critical DistinctionPreprint
Greene, A. J., Prepscius, C., & Levy, W. B. (2000) · Learning & Memory · doi:10.1101/lm.7.1.48 · Cited by 34 · see citing works
Tavakoli, S., Brunnstrom, K. E., Gutiérrez, J., et al. (2015) · Signal Processing Image Communication · doi:10.1016/j.image.2015.05.001 · Cited by 32 · see citing works
NR objective continuous video quality assessment model based on frame quality measure
Kawayoke, Y., & Horita, Y. (2008) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2008.4711772 · Cited by 30 · see citing works
Predicting subjective video quality from separated spatial and temporal assessment
Pastrana-Vidal, R. R., Gicquel, J. C., Blin, J. L., et al. (2006) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.642952 · Cited by 29 · see citing works
Delivery quality score model for Internet video
Yeganeh, H., Kordasiewicz, R. C., Gallant, M., et al. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7025402 · Cited by 24 · see citing works
Perceptual experience of time-varying video quality
Rehman, A., & Wang, Z. (2013) · International Workshop on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2013.6603240 · Cited by 22 · see citing works
An Augmented Autoregressive Approach to HTTP Video Stream Quality PredictionPreprint
Bampis, C. G., & Bovik, A. C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02709 · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content
Tu, Z., Wang, Y., Birkbeck, N., Adsumilli, B., & Bovik, A. C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3072221 · Cited by 363 · see citing works
“Temporal pooling is one of the most important, unresolved problems for video quality prediction [42], [60], [76], [82], [83].”
BackgroundSelf-citation
- Authors
- Bampis, C. G., Li, Z., Katsavounidis, I., & Bovik, A. C.
- Affiliations
- The University of Texas at Austin (US)
- Netflix (United States) (US)
- Venue
- IEEE Transactions on Image Processing
- Published
- 2018-07
- Volume
- 27
- Issue
- 7
- Pages
- 3316-3331
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W2793889002
- Semantic Scholar
- 150a7ec902c6d0cc3ccd25818bf7a6972d28413f
- Cited by
- 84
- References
- 70
- Citations by year
- 2026: 5 · 2025: 7 · 2024: 7 · 2023: 10 · 2022: 10 · 2021: 13 · 2020: 12 · 2019: 12 · 2018: 8
- FWCI
- 5.63
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Image Processing (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Netflix Inc.
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Traffic and Congestion Control
- Social Sciences › Sociology and Political Science › Multimedia Communication and Technology
- Keywords
- recurrent neural networks, autoregressive models, rebuffering, compression artifacts, time series forecasting, ensemble forecasting, low-bandwidth networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
