The DOI matches its Crossref record; 94 of the 94 references it cites were found in a registry or an index; 10 of the 27 sources its claims cite were checked for support.
UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content
Open accessAbstract
Recent years have witnessed an explosion of user-generated content (UGC) videos shared and streamed over the Internet, thanks to the evolution of affordable and reliable consumer capture devices, and the tremendous popularity of social media platforms. Accordingly, there is a great need for accurate video quality assessment (VQA) models for UGC/consumer videos to monitor, control, and optimize this vast content. Blind quality prediction of in-the-wild videos is quite challenging, since the quality degradations of UGC videos are unpredictable, complicated, and often commingled. Here we contribute to advancing the UGC-VQA problem by conducting a comprehensive evaluation of leading no-reference/blind VQA (BVQA) features and models on a fixed evaluation architecture, yielding new empirical insights on both subjective video quality studies and objective VQA model design. By employing a feature selection strategy on top of efficient BVQA models, we are able to extract 60 out of 763 statistical features used in existing methods to create a new fusion-based model, which we dub the VIDeo quality EVALuator (VIDEVAL), that effectively balances the trade-off between VQA performance and efficiency. Our experimental results show that VIDEVAL achieves state-of-the-art performance at considerably lower computational cost than other leading models. Our study protocol also defines a reliable benchmark for the UGC-VQA problem, which we believe will facilitate further research on deep learning-based VQA modeling, as well as perceptually-optimized efficient UGC video processing, transcoding, and streaming. To promote reproducible research and public evaluation, an implementation of VIDEVAL has been made available online: https://github.com/vztu/VIDEVAL.
Overview
CoChat Summary
TL;DR
This paper benchmarks blind video quality assessment (BVQA) models on large-scale user-generated content (UGC) video databases and proposes VIDEVAL, a feature-fusion model selected from 60 of 763 statistical features. VIDEVAL achieves state-of-the-art or near-state-of-the-art performance at lower computational cost than leading models, and the study provides a reproducible benchmark for UGC-VQA.
Key findings
- VIDEVAL selected 60 of 763 features and achieved SOTA/near-SOTA on all test sets: All-Combinedc SRCC 0.7960, PLCC 0.7939; KoNViD-1k SRCC 0.7832, PLCC 0.7803; LIVE-VQC SRCC 0.7522, PLCC 0.7514; YouTube-UGC SRCC 0.7787, PLCC 0.7733.
- TLVQM outperformed other BVQA models on LIVE-VQC by a notable margin (SRCC 0.7988, PLCC 0.8025) but underperformed on YouTube-UGC (SRCC 0.6693, PLCC 0.6590); models containing temporal features excelled on LIVE-VQC.
- Completely blind NIQE, ILNIQE, and VIIDEO performed poorly; on All-Combinedc, NIQE had SRCC 0.4622/PLCC 0.4773 and VIIDEO had SRCC 0.1039/PLCC 0.1621.
- Deep CNN baselines transferred well: ResNet-50 on KoNViD-1k had SRCC 0.8018/PLCC 0.8104; ensembling VIDEVAL with deep features improved performance by up to ∼4%, e.g., VIDEVAL+ResNet-50 on All-Combinedc had SRCC 0.8115/PLCC 0.8286.
Methods
- Evaluated leading BVQA/BIQA models on KoNViD-1k (1,200 videos), LIVE-VQC (585), YouTube-UGC (1,380; 1,323 color videos as YouTube-UGCc), and All-Combinedc (3,108) after INLSA MOS calibration; used SVR regression, 80%/20% train-test splits repeated 100 times, and reported SRCC, KRCC, PLCC, and RMSE.
- VIDEVAL was built from an initial 763-dimensional feature set drawn from BRISQUE, GM-LOG, HIGRADE-GRAD, FRIQUEE variants, and TLVQM; feature selection used random forest, linear SVM, and SFFS, choosing SVM importance with k=60 over 100 random splits; spatial features were computed every second frame within 1-second chunks with average and standard deviation pooling, then average pooled across chunks.
- Additional analyses included categorical subsets by resolution, content, and quality; cross-dataset generalization; eleven temporal pooling methods; complexity/runtime comparisons; and ensembling VIDEVAL with deep features.
Limitations
- Paper-stated: MOS standard deviation analysis is not presented because LIVE-VQC does not provide it; YouTube-UGC contains 57 grayscale videos that cause numerical errors for FRIQUEE, so color models were evaluated on a 1,323-video subset; temporal pooling results are only self-consistent and not comparable to prior experiments; not enough samples (8–20) per video were available for statistically meaningful pooling; non-natural/computer-generated content remains hard and lacks sufficiently large UGC-VQA datasets.
- Evident from the text: the benchmark covers three databases and a calibrated composite; categorical subsets for animation (81), screen content (163), and gaming (209) are small; cross-dataset generalization is modest (best SRCC values in Tables IX/X range roughly 0.33–0.70); and the deep models evaluated are mostly image-pretrained or picture-quality models rather than video-trained temporal CNNs.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 1 Not valid · 9 Not checked · 69 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These two databases, however, only modeled (camera) capture distortions on small numbers of not very diverse unique contents.”
What the sources say
- Partly supportedcites [12] [12] M. Nuutinen, T. Virtanen, M. Vaahteranoksa, T. Vuori, P. Oittinen, and J. H¨ akkinen, 'CVD2014-a database for evaluating no-reference video quality assessment algorithms,' IEEE Trans. Image Process. , vol. 25, no. 7, pp. 3073-3086, 2016. - sourceExperimental — accuracy not yet measured
Our copy supports the CVD2014 half only in part: the Abstract/Introduction and the repository Description state that CVD2014 models real camera-acquisition distortions ('the videos in the CVD2014 database are captured using 78 different cameras, and the distortions are related to the video acquisition process') on a small content base of 234 videos from five different scenes, and Section II.A notes the sequences had to be 'shot by different cameras and still be as similar as possible'. However, the paper never characterizes its own content as 'not very diverse' — Section II.A instead says the sequences 'were captured from many different scenes' — and it makes no evaluative claim of limited unique content. Nothing in our copy addresses LIVE-Qualcomm at all: Table I lists only LIVE [11] and LIVE Mobile [12], both with compression and transmission-error distortions, not capture distortions, so that half of the sentence finds no support here. (LLM, conf=0.70)
- Not checked — no full text of the sourcecites [13] [13] D. Ghadiyaram, J. Pan, A. C. Bovik, A. K. Moorthy, P. Panda, and K.-C. Yang, 'In-capture mobile video distortions: A study of subjective behavior and objective algorithms,' IEEE Trans. Circuits Syst. Video Technol. , vol. 28, no. 9, pp. 2061-2077, 2017. - source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Indeed, if temporal distortions (like judder) are present, they may severely underperform if the frame quality is high [81].”
What the sources say
- Supportedcites [81] [81] P. C. Madhusudana, X. Yu, N. Birkbeck, Y. Wang, B. Adsumilli, and A. C. Bovik, 'Subjective and objective quality assessment of high frame rate videos,' arXiv preprint arXiv:2007.11634 , 2020. - source
The Table 6 discussion says that when high spatial quality is combined with low temporal quality that a model does not assess, spatial measurements unduly influence its video-quality prediction. It also reports that VMAF fails to capture judder/stutter artifacts. This supports the claim for models that inadequately account for temporal distortions. (LLM, conf=0.91)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The most probable reason would be that these models were trained on picture quality datasets [64], [70], which contain different types of (strictly spatial) distortions than UGC-VQA databases.”
What the sources say
- Partly supportedcites [64] [64] Z. Ying, H. Niu, P. Gupta, D. Mahajan, D. Ghadiyaram, and A. Bovik, 'From patches to pictures (PaQ-2-PiQ): Mapping the perceptual space of picture quality,' in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2020. - sourceExperimental — accuracy not yet measured
Our copy supports the identification half of the attribution: PaQ-2-PiQ's database is a still-picture quality database whose contents are impaired by 'complex mixtures of highly diverse distortions' (Table 1 caption; Sec. 2 'Image Quality Datasets'; Sec. 3.1 UGC-like picture sampling). What the source does not support is the comparative/explanatory part of the claim: nowhere does our copy contrast these distortion types with those of UGC video (VQA) databases, characterise its distortions as 'strictly spatial' in opposition to temporal video distortions, or offer any account of why a picture-quality-trained model performs poorly on video. Searches for video/temporal-distortion comparisons returned only passing mentions (streaming services, VMAF, 'pictures and videos' in Sec. 5). (LLM, conf=0.70)
- Supportedcites [70] [70] V. Hosu, H. Lin, T. Sziranyi, and D. Saupe, 'Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment,' IEEE Trans. Image Process. , vol. 29, pp. 4041-4056, 2020. - source
The citation identifies a picture-quality dataset, and the Introduction (p. 1) identifies KonIQ-10k as a database of images with quality ratings. The Subjective Image Quality Assessment section (p. 4) lists the image degradations considered in those ratings. The proposed reason for the models’ poor performance is the citing authors’ inference, not a finding attributed to this source. (LLM, conf=0.90)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The empirical results show that a simple aggregation of these known models can achieve state-of-the-art (SOTA) performance.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Performance results are shown in Table VIII, wherein VIDEVAL still outperformed the other BVQA models on both low and high quality partitions.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
43 more claims
Create a free account to see all 86 claims and to verify the 6 references still open.
- Probably not a substantial claim. 24
- It may state two separate claims as one. 20
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ F. Complexity Analysis and Runtime Comparison · p. 13
“Deep ConvNet features ranked second in dimension (VGG-19 (4,080) and ResNet-50 (2,048)).”
§ A. Evaluation Protocol · p. 8
“The fully-connected layer (4,096-dim) from VGG-19 and average-pooled layer (2,048-dim) from ResNet50 served as deep feature descriptors”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ C. Performance Evaluation on Categorical Subsets · p. 11
“The proposed VIDEVAL model outperformed over all categories, followed by ResNet-50 and FRIQUEE, suggesting that VIDEVAL features are robust quality indicatives across different content categories.”
§ B. Performance on Individual and Combined Datasets · p. 10
“| ResNet-50 | 0.6199 | 0.6676 | 0.2781 | 0.4871 | 0.6378 | 0.6779 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Number of ratings | 136,800 (114 votes/video) | 205,000 (240 votes/video)”
Reported 240 · recomputed 350.42735
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content · p. 1
“Our experimental results show that VIDEVAL achieves state-of-theart performance at considerably lower computational cost than other leading models.”
§ F. Complexity Analysis and Runtime Comparison · p. 13
“| TLVQM | MATLAB-CPU | 183.8 | | VIDEVAL | MATLAB-CPU | 305.8 |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 3×
“To conquer the limits of data scale, they either propose to conduct patch-wise training [62], [63], [66] using global scores, or by pretraining deep nets on ImageNet [67], then fine tuning.”
MethodImage quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · cited 2×
“Some popular FR-VQA models, including PSNR, SSIM [25], and VMAF [21] have already been successfully and massively deployed to optimize streaming and shared/uploaded video encoding protocols by leading video service providers.”
MethodSelf-citationLIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · cited 1×
“As recommended in [80], we optimized the SVR parameter values (C, γ ) by a grid-search of 10×10 exponentially growing sequences (in our experiments, we used a grid of C = 21, 22, .”
MethodMaking a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766 · cited 8×
“before (Figure 10a) and after (Figure 10b) INLSA linear mapping, calibrated by NIQE [53] scores.”
MethodInfluentialSelf-citationMultiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253
Self-citationNo-Reference Image Quality Assessment in the Spatial Domain
Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2214050 · Cited by 6,064 · cited 7×
“We also avoid using duplicate features in different models, such as the BRISQUE-like features in HIGRADE, and the C-DIIVINE features in V-BLIINDS.”
MethodInfluentialSelf-citationDeep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196
Image information and visual quality
Sheikh, H. R., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.859378 · Cited by 4,104 · cited 1×
“These natural scene statistics (NSS) models predictably deviate in the presence of distortions, thereby characterizing perceived quality degradations [47].”
BackgroundSelf-citation
Research works printed but not linked: 4
- li, 2016, Toward A Practical Perceptual Video Quality Metric
- van der maaten, 2008, Visualizing data using t-SNE, J Mach Learn Res, 9, 2579
- zhang, 2011, IVP subjective quality video database
- 2010, VQEG HDTV Phase I Database
Cited by
VBench: Comprehensive Benchmark Suite for Video Generative Models
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02060 · Cited by 407
“Meanwhile, the Video Quality Assessment (VQA) methods [63, 94, 108–114] are primarily designed for real videos, thereby neglecting the unique challenges posed by generative models, such as artifacts in synthesized videos.”
MethodSelf-citationFAST-VQA: Efficient End-to-End Video Quality Assessment with Fragment Sampling
Wu, H., Chen, C., Hou, J., Liao, L., Wang, A., Sun, W., Yan, Q., & Lin, W. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20068-7_31 · Cited by 244
RAPIQUE: Rapid and Accurate Video Quality Prediction of User Generated Content
Tu, Z., Yu, X., Wang, Y., Birkbeck, N., Adsumilli, B., & Bovik, A. C. (2021) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2021.3090333 · Cited by 215
“Among these, BRISQUE [8], GM-LOG [11], FRIQUEE [12], V-BLIINDS [9], and VIDEVAL [6] have leveraged different sets of natu-ral scene statistics (NSS)-based quality-aware features, using them to train shallow regressors to predict subjective quality scores.”
BackgroundMethodInfluentialSelf-citationWu, H., Zhang, E., Liao, L., Chen, C., Hou, J., Wang, A., Sun, W., Yan, Q., & Lin, W. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01843 · Cited by 215
“Recent Video Quality Assessment (VQA) studies have gathered enormous human quality opinions [1–5] on in-the-wild user-generated contents (UGC) and attempted to use machine algorithms [6–8] to learn and predict these opinions, known as the UGCVQA problem [9].”
BackgroundInfluentialPerceptual video quality assessment: a survey
Min, X., Duan, H., Sun, W., Zhu, Y., & Zhai, G. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4133-3 · Cited by 218
Metadata
- Authors
- Tu, Z., Wang, Y., Birkbeck, N., Adsumilli, B., & Bovik, A. C.
- Affiliations
- The University of Texas at Austin (US)
- Google (United States) (US)
- Venue
- IEEE Transactions on Image Processing
- Published
- 2021
- Volume
- 30
- Pages
- 4449-4464
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W3030701471
- Semantic Scholar
- a0d2b46a4ee9eca6f32df1c6f6bf7b1fbe6c0e0e
- Cited by
- 363
- References
- 81
- Citations by year
- 2026: 40 · 2025: 75 · 2024: 84 · 2023: 74 · 2022: 46 · 2021: 36
- FWCI
- 21.56
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Image Processing (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Keywords
- blind video quality assessment, user-generated content, benchmark, feature selection, feature fusion, computational efficiency, perceptual video quality, in-the-wild videos, video streaming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content. https://app.cochat.ai/papers/10.1109/tip.2021.3072221
- The paper
- Tu, Z., Wang, Y., Birkbeck, N., Adsumilli, B., & Bovik, A. C. (2021). UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content. IEEE Transactions on Image Processing. https://doi.org/10.1109/tip.2021.3072221
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.