The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Quality Assessment of In-the-Wild Videos
Open accessAbstract
Quality assessment of in-the-wild videos is a challenging problem because of the absence of reference videos and shooting distortions. Knowledge of the human visual system can help establish methods for objective quality assessment of in-the-wild videos. In this work, we show two eminent effects of the human visual system, namely, content-dependency and temporal-memory effects, could be used for this purpose. We propose an objective no-reference video quality assessment method by integrating both effects into a deep neural network. For content-dependency, we extract features from a pre-trained image classification neural network for its inherent content-aware property. For temporal-memory effects, long-term dependencies, especially the temporal hysteresis, are integrated into the network with a gated recurrent unit and a subjectively-inspired temporal pooling layer. To validate the performance of our method, experiments are conducted on three publicly available in-the-wild video quality assessment databases: KoNViD-1k, CVD2014, and LIVE-Qualcomm, respectively. Experimental results demonstrate that our proposed method outperforms five state-of-the-art methods by a large margin, specifically, 12.39%, 15.71%, 15.45%, and 18.09% overall performance improvements over the second-best method VBLIINDS, in terms of SROCC, KROCC, PLCC and RMSE, respectively. Moreover, the ablation study verifies the crucial role of both the content-aware features and the modeling of temporal-memory effects. The PyTorch implementation of our method is released at https://github.com/lidq92/VSFA.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These SOTA deep CNN-based BVQA models [71][74] produce accurate quality predictions on legacy (single synthetic distortion) video datasets [1], [6], but struggle on recent in-the-wild UGC databases [10], [12], [13].”
From UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content (2021) · cites this paper as [74]
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Making a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766
No-Reference Image Quality Assessment in the Spatial Domain
Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2214050 · Cited by 6,064
Study of Subjective and Objective Quality Assessment of Video
Seshadrinathan, K., Soundararajan, R., Bovik, A. C., & Cormack, L. K. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2042111 · Cited by 1,160
Video quality assessment based on structural distortion measurement
Wang, Z., Lu, L., & Bovik, A. C. (2004) · Signal Processing: Image Communication · doi:10.1016/s0923-5965(03)00076-6 · Cited by 1,024
A Survey on Quality of Experience of HTTP Adaptive Streaming
Seufert, M., Egger, S., Slanina, M., Zinner, T., Hoßfeld, T., & Tran-Gia, P. (2015) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2014.2360940 · Cited by 844
Unsupervised feature learning framework for no-reference image quality assessment
Ye, P., Kumar, J., Kang, L., & Doermann, D. S. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247789 · Cited by 833
Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos
Seshadrinathan, K., & Bovik, A. C. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2034992 · Cited by 708 · cited 2×
“Traditional VQAmethods consider structures [47, 48], gradients [21], motion [22, 36], energy [18], saliency [52, 54], or natural video statistics [9, 28, 35, 57].”
BackgroundMethodBlind Prediction of Natural Video Quality
Saad, M. A., Bovik, A. C., & Charrier, C. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2299154 · Cited by 471 · cited 5×
“content and may suffer from complex mixed real-world distortions that are temporally heterogeneous. On account of this, current state-of-the-art video quality assessment (VQA) methods (e.g., VBLIINDS [35] and VIIDEO [28]) validated on traditional synthetic VQA databases [30, 38] fail in predicting the quality of in-th…”
MethodInfluentialA Completely Blind Video Integrity Oracle
Mittal, A., Saad, M. A., & Bovik, A. C. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2502725 · Cited by 357
Large-Scale Study of Perceptual Video Quality
Sinno, Z., & Bovik, A. C. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2869673 · Cited by 310
Video Quality Assessment on Mobile Devices: Subjective, Behavioral and Objective Studies
Moorthy, A. K., Choi, L. K., Bovik, A. C., & Veciana, G. A. D. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2212417 · Cited by 308
Measurement of Quality of Experience of Video-on-Demand Services: A Survey
Juluri, P., Tamarapalli, V., & Medhi, D. (2016) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2015.2401424 · Cited by 229
Study of Temporal Effects on Subjective Video Quality of Experience
Bampis, C. G., Li, Z., Moorthy, A. K., Katsavounidis, I. X., Aaron, A., & Bovik, A. C. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2729891 · Cited by 188
ViS3: an algorithm for video quality assessment via analysis of spatial and spatiotemporal slices
Vu, P. V., & Chandler, D. M. (2014) · Journal of Electronic Imaging · doi:10.1117/1.jei.23.1.013016 · Cited by 185 · cited 1×
“vious methods, we consider using GRU network to model the long-term dependencies for feature integration. In the quality pooling aspect, the simple average pooling strategy is adopted by many methods [20, 28, 36, 45, 57]. Several pooling strategies considering the recency effect or the worst quality section influence…”
MethodVideoSet: A large-scale compressed video quality dataset based on JND measurement
Wang, H., Katsavounidis, I. X., Zhou, J., Park, J.-H., Lei, S., Zhou, X., Pun, M.-O., Jin, X., Wang, R., Wang, X., et al. (2017) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2017.04.009 · Cited by 177
Spatiotemporal Statistics for Video Quality Assessment
Li, X., Guo, Q., & Lu, X. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2568752 · Cited by 176
Which Has Better Visual Quality: The Clear Blue Sky or a Blurry Animal?
Li, D., Jiang, T., Lin, W., & Jiang, M. F. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2875354 · Cited by 156
Self-citationIn-Capture Mobile Video Distortions: A Study of Subjective Behavior and Objective Algorithms
Ghadiyaram, D., Pan, J., Bovik, A. C., Moorthy, A. K., Panda, P., & Yang, K.-C. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2707479 · Cited by 154
Temporal hysteresis model of time varying subjective video quality
Seshadrinathan, K., & Bovik, A. C. (2011) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2011.5946613 · Cited by 137
End-to-End Blind Quality Assessment of Compressed Videos Using Deep Neural Networks
Liu, W., Duanmu, Z., & Wang, Z. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240643 · Cited by 126
Kim, W., Kim, J., Ahn, S., Kim, J., & Lee, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_14 · Cited by 125
No-Reference Video Quality Assessment With 3D Shearlet Transform and Convolutional Neural Networks
Li, Y., Po, L.-M., Cheung, T. C.-H., Xu, X., Feng, L., Fang, Y., & Cheung, K. W. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2430711 · Cited by 121
Video Quality Pooling Adaptive to Perceptual Distortion Severity
Park, J., Seshadrinathan, K., Lee, S., & Bovik, A. C. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2219551 · Cited by 119
Attention Driven Foveated Video Quality Assessment
You, J., Ebrahimi, T., & Perkis, A. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2287611 · Cited by 59
Study of Saliency in Objective Video Quality Assessment
Zhang, W., & Liu, H. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2651410 · Cited by 50
Novel Spatio-Temporal Structural Information Based Video Quality Metric
Wang, Y., Jiang, T., Ma, S., & Gao, W. (2012) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2012.2186745 · Cited by 45
Self-citationVideo quality assessment accounting for temporal visual masking of local flicker
Choi, L. K., & Bovik, A. C. (2018) · Signal Processing Image Communication · doi:10.1016/j.image.2018.06.009 · Cited by 34
Quality-of-Experience of Adaptive Video Streaming
Duanmu, Z., Ma, K., & Wang, Z. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123418 · Cited by 33
An optical flow-based no-reference video quality assessment algorithm
Manasa, K., & Channappayya, S. S. (2016) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2016.7532789 · Cited by 31
Blind image quality assessment with hierarchy: Degradation from local structure to deep semantics
Wu, J., Zeng, J., Dong, W., Shi, G., & Lin, W. (2018) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2018.12.005 · Cited by 29
Semantic-aware blind image quality assessment
Siahaan, E., Hanjalić, A., & Redi, J. (2017) · Signal Processing Image Communication · doi:10.1016/j.image.2017.10.009 · Cited by 28
Spatiotemporal Feature Combination Model for No-Reference Video Quality Assessment
Men, H., Lin, H., & Saupe, D. (2018) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2018.8463426 · Cited by 27
Triantaphillidou, S., Allen, E. M., & Jacobson, R. E. (2007) · Journal of Imaging Science and Technology · doi:10.2352/j.imagingsci.technol.(2007)51:3(259 · Cited by 26
Using multiple spatio-temporal features to estimate video quality
Freitas, P. G., Akamine, W. Y. L., & Farias, M. C. Q. (2018) · Signal Processing Image Communication · doi:10.1016/j.image.2018.02.010 · Cited by 22
Evaluating the Role of Content in Subjective Video Quality Assessment
Mirković, M., Vrgović, P., Culibrk, D. R., Stefanović, D., & Anderla, A. (2014) · The Scientific World JOURNAL · doi:10.1155/2014/625219 · Cited by 22
A no-reference video quality predictor for compression and scaling artifacts
Ghadiyaram, D., Chen, C., Inguva, S., & Kokaram, A. C. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296922 · Cited by 21
Empirical evaluation of no-reference VQA methods on a natural video quality database
Men, H., Lin, H., & Saupe, D. (2017) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2017.7965644 · Cited by 18
A spatiotemporal model of video quality assessment via 3D gradient differencing
Lu, W. B., He, R., Yang, J., Jia, C., & Gao, X. (2018) · Information Sciences · doi:10.1016/j.ins.2018.11.003 · Cited by 18
Content-aware objective video quality assessment
Ortiz-Jaramillo, B., Niño-Castañeda, J. O., Platiša, L., & Philips, W. R. (2016) · Journal of Electronic Imaging · doi:10.1117/1.jei.25.1.013011 · Cited by 15
Psychometric scaling of TID2013 dataset
Mikhailiuk, A., Pérez‐Ortiz, M., & Mantiuk, R. (2018) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2018.8463376 · Cited by 11
Research works printed but not linked: 14
- Cho Kyunghyun, 2014, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.
- Dodge Samuel, Understanding how image quality affects deep neural networks
- Fan Lijie, 2018, Boqing Gong, and Junzhou Huang.
- Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770--778. Kaiming He Xiangyu Zhang Shaoqing Ren and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. 770--778.
- Hosu Vlad, The Konstanz natural video database (KoNViD-1k)
- Kingma Diederik P, 2014, Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 · arXiv
- Nuutinen Mikko, 2016, CVD2014--A database for evaluating no-reference video quality assessment algorithms. IEEE TIP, 25
- Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS-W. Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS-W.
- Rimac-Drlje Snjezana, Influence of temporal pooling method on the objective video quality evaluation
- Seufert Michael, A comparison of temporal pooling methods for HTTP adaptive video streaming
- Xu Jingtao, No-reference video quality assessment via feature learning
- Zhang Yu, 2018, Blind Video Quality Assessment with Weakly Supervised Learning and Resampling Strategy. TCSVT
- Zhang Yu, 2019, Objective Video Quality Assessment Combining Transfer Learning With CNN. TNNLS
- Zhu Yun, Blind video quality assessment based on spatio-temporal internal generative mechanism
Cited by
Sorted by citations
UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content
Tu, Z., Wang, Y., Birkbeck, N., Adsumilli, B., & Bovik, A. C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3072221 · Cited by 363
“nnected layers to predict video quality. Zhang [69] leveraged transfer learning to develope a general-purpose BVQA framework based on weakly supervised learning and a resampling strategy. In the VSFA [70] model, the authors applied a pre-trained image classification CNN as a deep feature extractor and integrated the fr…”
MethodFAST-VQA: Efficient End-to-End Video Quality Assessment with Fragment Sampling
Wu, H., Chen, C., Hou, J., Liao, L., Wang, A., Sun, W., Yan, Q., & Lin, W. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20068-7_31 · Cited by 286
A Deep Learning based No-reference Quality Assessment Model for UGC Videos
Sun, W., Min, X., Lu, W., & Zhai, G. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548329 · Cited by 267
X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval
Gorti, S. K., Vouitsis, N., Ma, J., Golestan, K., Volkovs, M., Garg, A., & Yu, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00495 · Cited by 261
RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content
Tu, Z., Yu, X., Wang, Y., Birkbeck, N., Adsumilli, B., & Bovik, A. C. (2021) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2021.3090333 · Cited by 235
Metadata
- Authors
- Li, D., Jiang, T., & Jiang, M.
- Affiliations
- Peking University (CN)
- Venue
- Proceedings of the 27th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2019-10-15
- Pages
- 2351-2359
- OpenAlex
- W2965644659
- Cited by
- 387
- References
- 41
- Citations by year
- 2026: 44 · 2025: 80 · 2024: 76 · 2023: 76 · 2022: 52 · 2021: 49 · 2020: 10
- FWCI
- 9.05
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- the National Basic Research Program of China (973 Program) (2015CB351803)
- the Natural Science Foundation of China (61572042, 61520106004, 61527804)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- no-reference video quality assessment, human visual system, temporal memory, gated recurrent unit, temporal pooling, long-range dependencies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
