CoChat
Sign Up
Cited by 310Image and Video Quality AssessmentImage Retrieval and Classification TechniquesVideo Coding and Compression Technologies

Large-Scale Study of Perceptual Video Quality

Open access
Sinno, Zeina, Bovik, Alan Conrad·2019·IEEE Transactions on Image Processing
DOI 10.1109/tip.2018.2869673
Open this paper in CoChat — free accountOpen PDF

Abstract

The great variations of videographic skills in videography, camera designs, compression and processing protocols, communication and bandwidth environments, and displays leads to an enormous variety of video impairments. Current noreference (NR) video quality models are unable to handle this diversity of distortions. This is true in part because available video quality assessment databases contain very limited content, fixed resolutions, were captured using a small number of camera devices by a few videographers and have been subjected to a modest number of distortions. As such, these databases fail to adequately represent real world videos, which contain very different kinds of content obtained under highly diverse imaging conditions and are subject to authentic, complex and often commingled distortions that are difficult or impossible to simulate. As a result, NR video quality predictors tested on real-world video data often perform poorly. Towards advancing NR video quality prediction, we have constructed a largescale video quality assessment database containing 585 videos of unique content, captured by a large number of users, with wide ranges of levels of complex, authentic distortions. We collected a large number of subjective video quality scores via crowdsourcing. A total of 4776 unique participants took part in the study, yielding more than 205000 opinion scores, resulting in an average of 240 recorded human opinions per video. We demonstrate the value of the new resource, which we call the LIVE Video Quality Challenge Database (LIVE-VQC for short), by conducting a comparison of leading NR video quality predictors on it. This study is the largest video quality assessment study ever conducted along several key dimensions: number of unique contents, capture devices, distortion types and combinations of distortions, study participants, and recorded subjective scores. The database is available for download on this link: http://live.ece.utexas.edu/research/LIVEVQC/index.html.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tip.2018.2869673

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LIVE-VQC is a large-scale UGC-VQA database with 585 videos, crowdsourced on Amazon Mechanical Turk to collect human opinions from 4,776 unique participants.

    “LIVE-VQC [9] was another large-scale UGC-VQA database with 585 videos, crowdsourced on Amazon Mechanical Turk to collect human opinions from 4,776 unique participants.”

    From UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content (2021) · cites this paper as [9]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 39 · Cited by 310

Research works printed but not linked: 34

  1. [5][5] K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. K. Cormack, 'Study of subjective and objective quality assessment of video,' IEEE Trans. Image Process. , vol. 19, no. 6, pp. 1427-1441, 2010. -
  2. [6][6] F. De Simone, M. Tagliasacchi, M. Naccari, S. Tubaro, and T. Ebrahimi, 'A H. 264/AVC video database for the evaluation of quality metrics,' Intern. Conf. Acous. Sp. Sign. Process. (ICASSP) , pp. 2430-2433, 2010. -
  3. [7][7] C. Chen, L. K. Choi, G. de Veciana, C. Caramanis, R. W. Heath, and A. Bovik, 'Modeling the time varying subjective quality of HTTP video streams with rate adaptations,' IEEE Trans. on Image Process. , vol. 23, no. 5, pp. 2206-2221, 2014. -
  4. [8][8] A. K. Moorthy, L. K. Choi, A. C. Bovik, and G. De Veciana, 'Video quality assessment on mobile devices: Subjective, behavioral and objective studies,' IEEE J. Select. Topics Sign. Process. , vol. 6, no. 6, pp. 652-671, 2012. -
  5. [9][9] C. Keimel, J. Habigt, T. Habigt, M. Rothbucher, and K. Diepold, 'Visual quality of current coding technologies at high definition IPTV bitrates,' IEEE Int'l Wkshp. Multidim. Sign. Process. , pp. 390-393, 2010. -
  6. [10][10] C. Keimel, A. Redl, and K. Diepold, 'The TUM high definition video datasets,' Int'l Wkshp. Qual. Multim. Exper. , pp. 97-102, 2012. -
  7. [11][11] J. Y. Lin, R. Song, C.-H. Wu, T. Liu, H. Wang, and C.-C. J. Kuo, 'MCLV: A streaming video quality assessment database,' J. Vis. Commun. Image Repres. , vol. 30, pp. 1-9, 2015. -
  8. [12][12] H. Wang, I. Katsavounidis, J. Zhou, J. Park, S. Lei, X. Zhou, M.-O. Pun, X. Jin, R. Wang, X. Wang et al. , 'Videoset: A large-scale compressed video quality dataset based on jnd measurement,' J. of Vis. Comm. Im. Rep. , vol. 46, pp. 292-302, 2017. -
  9. [13][13] D. Ghadiyaram, J. Pan, A. C. Bovik, A. K. Moorthy, P. Panda, and K.C. Yang, 'In-capture mobile video distortions: A study of subjective behavior and objective algorithms,' IEEE Trans. Circ. Syst. Video Technol. , 2018. -
  10. [14][14] M. Nuutinen, T. Virtanen, M. Vaahteranoksa, T. Vuori, P. Oittinen, and J. H¨ akkinen, 'CVD2014 a database for evaluating no-reference video quality assessment algorithms,' IEEE Trans. Image Process. , vol. 25, no. 7, pp. 3073-3086, 2016. -
  11. [15][15] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al. , 'Imagenet large scale visual recognition challenge,' Intern. J. Comp. Vis. , vol. 115, no. 3, pp. 211-252, 2015. -
  12. [16][16] J. A. Redi, T. Hoßfeld, P. Korshunov, F. Mazza, I. Povoa, and C. Keimel, 'Crowdsourcing-based multimedia subjective evaluations: a case study on image recognizability and aesthetic appeal,' ACM Intern. Wkshp Crowds. Multim. , pp. 29-34, 2013. -
  13. [17][17] D. Ghadiyaram and A. C. Bovik, 'Massive online crowdsourced study of subjective and objective picture quality,' IEEE Trans. Image Process. , vol. 25, no. 1, pp. 372-387, 2016. -
  14. [18][18] K. T. Chen, C. J. Chang, C. C. Wu, Y. C. Chang, and C. L. Lei, 'Quadrant of euphoria: A crowdsourcing platform for QoE assessment,' IEEE Net. , vol. 24, no. 2, 2010. -
  15. [19][19] M. Seufert and T. Hoßfeld, 'One shot crowdtesting: Approaching the extremes of crowdsourced subjective quality testing,' Wkshp Perc. Qual. Sys. (PQS 2016) , pp. 122-126, 2016. -
  16. [20][20] C. Keimel, J. Habigt, C. Horch, and K. Diepold, 'Qualitycrowd a framework for crowd-based quality evaluation,' Pict. Cod. Symp. (PCS), 2012 , pp. 245-248, 2012. -
  17. [21][21] V. Hosu, F. Hahn, M. Jenadeleh, H. Lin, H. Men, T. Szir´ anyi, S. Li, and D. Saupe, 'The konstanz natural video database (konvid-1k),' Qual. Mult. Exp. (QoMEX) , pp. 1-6, 2017. -
  18. [23][23] T. Hossfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt, K. Diepold, and P. Tran-Gia, 'Best practices for QoE crowdtesting: QoE assessment with crowdsourcing,' IEEE Trans. Multim. , vol. 16, no. 2, pp. 541-558, 2014. -
  19. [24][24] ´ O. Figuerola Salas, V. Adzic, A. Shah, and H. Kalva, 'Assessing internet video quality using crowdsourcing,' Proc. ACM Int'; Wkshp. Crowd. Multim. , pp. 23-28, 2013. -
  20. [25][25] M. Shahid, J. Søgaard, J. Pokhrel, K. Brunnstr¨ om, K. Wang, S. Tavakoli, and N. Gracia, 'Crowdsourcing based subjective quality assessment of adaptive video streaming,' Wkshp. Qual. Multim. Exper. , pp. 53-54, 2014. -
  21. [26][26] Y. Chen, K. Wu, and Q. Zhang, 'From QoS to QoE: A tutorial on video quality assessment,' IEEE Comm. Surv. Tutorials , vol. 17, no. 2, pp. 1126-1165, 2015. -
  22. [27][27] B. Rainer and C. Timmerer, 'Quality of experience of web-based adaptive HTTP streaming clients in real-world environments using crowdsourcing,' ACM Wkshp. Desig. Quality Deployment Adapt. Video Streaming , pp. 19-24, 2014. -
  23. [31][31] Z. Sinno and A. C. Bovik, 'Large scale subjective video quality study,' in IEEE Intern. Conf. Image Process. (ICIP) , Oct. 2018, accepted. -
  24. [32][32] C. Vondrick, D. Patterson, and D. Ramanan, 'Efficiently scaling up crowdsourced video annotation,' Int. J. Comp. Vis. , vol. 101, no. 1, pp. 184-204, 2013. -
  25. [34][34] I. Abramov, J. Gordon, O. Feldman, and A. Chavarga, 'Sex and vision II: color appearance of monochromatic lights,' Biol. Sex Diff. , vol. 3, no. 1, p. 21, 2012. -
  26. [35][35] B. W. Rovner, R. J. Casten, and W. S. Tasman, 'Effect of depression on vision function in age-related macular degeneration,' Arch. Opht. , vol. 120, no. 8, pp. 1041-1044, 2002. -
  27. [36][36] A. Mittal, R. Soundararajan, and A. C. Bovik, 'Making a completely blind image quality analyzer,' IEEE Sign. Process, Lett. , vol. 20, no. 3, pp. 209-212, 2013. -
  28. [40][40] X. Li, Q. Guo, and X. Lu, 'Spatiotemporal statistics for video quality assessment,' IEEE Trans. Image Process. , vol. 25, no. 7, pp. 3329-3342, 2016. -
  29. [41][41] H. Men, H. Lin, and D. Saupe, 'Empirical evaluation of no-reference vqa methods on a natural video quality database,' Wkshp. Qual. Multim. Exper. , pp. 1-3, 2017. -
  30. [42][42] K. Zhu, C. Li, V. Asari, and D. Saupe, 'No-reference video quality assessment based on artifact measurement and statistical analysis,' IEEE Trans. on Circ. Sys. Vid. Tech. , vol. 25, no. 4, pp. 533-546, 2015. -
  31. [43][43] C. Wang, L. Su, and Q. Huang, 'Cnn-mr for no reference video quality assessment,' Inform. Sci. Contr. Eng. (ICISCE) , pp. 224-228, 2017. -
  32. [44][44] B. Sch¨ olkopf, A. J. Smola, R. C. Williamson, and P. L. Bartlett, 'New support vector algorithms,' Neur. Comp. , vol. 12, no. 5, pp. 1207-1245, 2000. -
  33. [46][46] M. Stone, 'Cross-validatory choice and assessment of statistical predictions,' J. Royal Stat. Soc. Ser. B (Methodol.) , pp. 111-147, 1974. -
  34. [47][47] 'Final report from the video quality experts group on the validation of objective models of video quality assessment.' Video Quality Expert Group (VQEG), 2000. <!-- image --> Zeina Sinno received the B.E. degree (Hons.) in electrical and computer engineering with a minor in mathematics from the American University of Beirut in 2013 and the M.S. in electrical and computer engineering from The University of Texas at Austin in 2015, where she is currently pursuing the Ph.D. degree with the Laboratory for Image and Video Engineering. Her research interests focus on image and video processing, and machine learning. <!-- image --> Alan Conrad Bovik (F95) received the B.S., M.S., and Ph.D. degrees in electrical and computer engineering from the University of Illinois in 1980, 1982, and 1984, respectively. He is currently the Cockrell Family Regents Endowed Chair Professor with The University of Texas at Austin. His books include the Handbook of Image and Video Processing, Modern Image Quality Assessment, and The Essential Guides to Image and Video Processing. His research interests are digital video, image processing, and visual perception. He was a recipient of the 2015 Primetime Emmy Award for outstanding achievement in engineering development from the Television Academy, the 2017 Edwin H. Land Medal from the Optical Society of America, the 2019 IEEE Fourier Award, and the Society Award from the IEEE Signal Processing Society. He has also received about 10 journal Best Paper Awards including the 2016 IEEE Signal Processing Society Sustained Impact Award. He also created/Chaired the IEEE International Conference on Image Processing, which was first held in Austin, TX, USA, 1994. He Co-Founded and was the longest-serving Editor-in-Chief of the IEEE Transactions on Image Processing.

Authors
Sinno, Z., & Bovik, A. C.
Affiliations
The University of Texas at Austin (US)
ORCID
Venue
IEEE Transactions on Image Processing
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2019-02
Volume
28
Issue
2
Pages
612-627
ISSN
  • 1057-7149 (ISSN-L)
  • 1941-0042
DOI
10.1109/tip.2018.2869673
OpenAlex
W2791258091
Cited by
310
References
39
Citations by year
2026: 27 · 2025: 53 · 2024: 58 · 2023: 53 · 2022: 45 · 2021: 49 · 2020: 14 · 2019: 10 · 2018: 1
FWCI
9.25
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Image Processing (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
  • Physical Sciences › Computer Science › Signal Processing › Video Coding and Compression Technologies
Keywords
no-reference video quality assessment, video quality prediction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review