Spatiotemporal Feature Integration and Model Fusion for Full Reference Video Quality Assessment
Open accessAbstract
The recently developed video multi-method assessment fusion (VMAF) framework integrates multiple quality-aware features to accurately predict the video quality. However, the VMAF does not yet exploit important principles of temporal perception that are relevant to the perceptual video distortion measurement. Here, we propose two improvements to the VMAF framework, called spatiotemporal VMAF and ensemble VMAF, which leverage perceptually-motivated space-time features that are efficiently calculated at multiple scales. We also conducted a large subjective video study, which we have found to be an excellent resource for training our feature-based approaches. In rigorous experiments, we found that the proposed algorithms demonstrate the state-of-the-art performance on multiple video applications. The compared algorithms will be made available as a part of the open source package in https://github.com/Netflix/vmaf.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892 · see citing works
Self-citationMultiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253 · see citing works
Self-citationNo-Reference Image Quality Assessment in the Spatial Domain
Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2214050 · Cited by 6,064 · see citing works
Self-citationA new standardized method for objectively measuring video quality
Pinson, M. H., & Wolf, S. (2004) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2004.834028 · Cited by 1,182 · cited 1× · see citing works
“VQM-VFD [12], [36] used a neural network trained with a large number of features such as edge features.”
MethodMotion Tuned Spatio-Temporal Quality Assessment of Natural Videos
Seshadrinathan, K., & Bovik, A. C. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2034992 · Cited by 708 · cited 3× · see citing works
“Video-based FR VQA models have also been studied in the literature [11]–[14].”
BackgroundMethodSelf-citationBlind Prediction of Natural Video Quality
Saad, M. A., Bovik, A. C., & Charrier, C. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2299154 · Cited by 471 · see citing works
Self-citationVideo Quality Assessment by Reduced Reference Spatio-Temporal Entropic Differencing
Soundararajan, R., & Bovik, A. C. (2013) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2012.2214933 · Cited by 368 · cited 7× · see citing works
“This local operation approximates the multi-scale multi-orientation steerable filter decomposition used in [20] and is very compute-efficient.”
BackgroundMethodInfluentialSelf-citationA Completely Blind Video Integrity Oracle
Mittal, A., Saad, M. A., & Bovik, A. C. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2502725 · Cited by 357 · cited 1× · see citing works
“NR models exploit distortion-specific or natural video statistical models to predict quality without using any reference information [22]–[30].”
BackgroundSelf-citationViS3: an algorithm for video quality assessment via analysis of spatial and spatiotemporal slices
Vu, P. V., & Chandler, D. M. (2014) · Journal of Electronic Imaging · doi:10.1117/1.jei.23.1.013016 · Cited by 185 · cited 1× · see citing works
“In the experiments, we relied on a wide variety of subjective video databases: LIVE VQA [54], LIVE Mobile [42] 2 , CSIQ-VQA [55], NFLX [15], SHVC [56] 3 , VQEG-HD3 [57], EPFL-Polimi [58], USC-JND [59], LIVE-NFLX [60] and LIVE-HTTP [61].”
MethodA spatiotemporal most-apparent-distortion model for video quality assessment
Vu, P. V., Vu, C. T., & Chandler, D. M. (2011) · 2011 18th IEEE International Conference on Image Processing · doi:10.1109/icip.2011.6116171 · Cited by 180 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Two-Level Approach for No-Reference Consumer Video Quality Assessment
Korhonen, J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2923051 · Cited by 354 · see citing works
- Authors
- Bampis, C. G., Li, Z., & Bovik, A. C.
- Affiliations
- Netflix (United States) (US)
- The University of Texas at Austin (US)
- Venue
- IEEE Transactions on Circuits and Systems for Video Technology
- Published
- 2019-08
- Volume
- 29
- Issue
- 8
- Pages
- 2256-2270
- ISSN
- 1051-8215 (ISSN-L)
- 1558-2205
- OpenAlex
- W2796677919
- Cited by
- 128
- References
- 10
- Citations by year
- 2026: 15 · 2025: 5 · 2024: 15 · 2023: 19 · 2022: 21 · 2021: 25 · 2020: 13 · 2019: 12 · 2018: 3
- FWCI
- 6.42
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Circuits and Systems for Video Technology (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Netflix Inc.
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Computer Science › Signal Processing › Video Coding and Compression Technologies
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- VMAF, model fusion, ensemble learning, time perception, multi-scale features, video quality prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
