Why no-reference metrics for image and video quality lack accuracy and reproducibility
Open accessThe DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index.
Abstract
This article provides a comprehensive overview of no reference (NR) metrics for image quality analysis (IQA) and video quality analysis (VQA). We examine 26 independent evaluations of NR metrics (previously published) and analyze 32 NR metrics on six IQA datasets and six VQA datasets (new results). Where NR metric developers claim Pearson correlation values between 0.66 and 0.99, our measurements range from 0.0 to 0.63. None of the NR metrics we analyzed are accurate enough to be deployed by industry. Performance evaluations that indicate otherwise are based on insufficient data and highly inaccurate. We will examine development strategies, tools, datasets, root cause analysis, and our baseline metric for collaboration, Sawatch.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similar limitations have been widely discussed in both full-reference and no-reference quality assessment, particularly when objective scores lack reproducibility or fail to generalize across content and distortion conditions [54].”
From A Survey and Tutorial on Image Quality Assessment with a Contrast-Weighted Structural Similarity Framework (2026) · cites this paper as [54]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Traditional full-reference metrics often rely on pixel-wise or structural differences that do not always align with human visual perception, particularly when images contain multiple distortion types, local contrast changes, or domain-specific degradation patterns.”
From A Survey and Tutorial on Image Quality Assessment with a Contrast-Weighted Structural Similarity Framework (2026) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398 · cited 2×
“LBP [29] was intended for texture classifi-cation.”
BackgroundMaking a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766 · cited 2×
“Retraining is particularly important for ML-NSS metrics, and MATLAB offers tools to re-train NIQE [34] and BRISQUE [10].”
MethodNo-Reference Image Quality Assessment in the Spatial Domain
Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2214050 · Cited by 6,064 · cited 2×
“BRISQUE [10] uses mean subtracted contrast normalized (MSCN) coefficients.”
MethodA Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms
Sheikh, H. R., Sabir, M. F., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.881959 · Cited by 2,966 · cited 3×
“Datasets with conventional experiment designs, like LIVE-2006 [70], avoid media with camera impairments.”
BackgroundMethodInfluentialBlind Image Quality Assessment: From Natural Scene Statistics to Perceptual Quality
Moorthy, A. K., & Bovik, A. C. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2147325 · Cited by 1,763
Blind Image Quality Assessment: A Natural Scene Statistics Approach in the DCT Domain
Saad, M. A., Bovik, A. C., & Charrier, C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2191563 · Cited by 1,751
A new standardized method for objectively measuring video quality
Pinson, M. H., & Wolf, S. (2004) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2004.834028 · Cited by 1,182 · cited 2×
“The best known of those are Video Quality Metric ( VQM ) [102] from ITU-T Rec.”
BackgroundSelf-citationTalebi, H., & Milanfar, P. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2831899 · Cited by 1,023 · cited 1×
“The NR metric with the best accuracy is NIMA [33] with ρ = 0 .”
BackgroundBlind image quality evaluation using perception based features
Venkatanath, N., Praneeth, D., Bh, M. C., Channappayya, S. S., & Medasani, S. S. (2015) · 2015 Twenty First National Conference on Communications (NCC) · doi:10.1109/ncc.2015.7084843 · Cited by 978 · cited 1×
“PIQE [41] takes inspiration from NIQE and BRISQUE , using both circularly-symmetric Gaussian weighting function and MSCN. SpEED-NR [44] uses a Gaussian scale mixture (GSM) model.”
MethodAVA: A large-scale database for aesthetic visual analysis
Murray, N., Marchesotti, L., & Perronnin, F. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247954 · Cited by 936
Research works printed but not linked: 12
- 2020, National Telecommunications and Information Administration, Institute for Telecommunication Sciences, NR Metric Framework
- Katsavounidis, 2018, Digital Optimizer—A Perceptual Video Encoding Optimization Framework.
- 2021, AGH Video Qualtiy of Experience (QoE Team) Indicators
- 2008, Subjective Video Quality Assessment Methods for Multimedia Applications
- 2021, VQET Image Quality Evaluation Tool (VIQET)
- Kahneman, 2021, Noise: A Flaw in Human Judgement
- 10.1007/s41233-019-0026-4 · DOI
- 2021, Laboratory for Image & Video Engineering
- 2021, The Konstanz Visual Quality Databases
- 2022, VQEG HDTV Test. Vqeghd1
- Lee, 2020, This Ugly T-Shirt Makes You Invisible to Facial Recognition Tech
- 2022, Sawatch Demo
Cited by
Deep neural network for blind visual quality assessment of 4K content
Lu, W., Sun, W., Min, X., Zhu, W., Zhou, Q., He, J., Wang, Q., Zhang, Z., Wang, T., & Zhai, G. (2023) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2022.3221689 · Cited by 60
Deep learning techniques for atmospheric turbulence removal: a review
Hill, P., Anantrasirichai, N., Achim, A., & Bull, D. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11086-6 · Cited by 43
Super-resolution image visual quality assessment based on structure–texture features
Zhou, F., Wei, S., Lu, Z., Kang, B., Chen, M., & Qiu, G. (2023) · Signal Processing Image Communication · doi:10.1016/j.image.2023.117025 · Cited by 20
Beyond Correlation: Evaluating Multimedia Quality Models With the Constrained Concordance Index
Ragano, A., Martinez, H. B., & Hines, A. (2024) · IEEE transactions on multimedia · doi:10.1109/tmm.2025.3542991 · Cited by 8
“However, with only 6 raters, this interval can expand to more than 1.5 [32].”
BackgroundInfluentialQuality measurement methods for video assisting refereeing systems
Brunnstrom, K. E., Djupsjöbacka, A., Ozoliņš, O., Billingham, J., Wistel, K., & Evans, N. (2023) · Sports Engineering · doi:10.1007/s12283-023-00408-6 · Cited by 8
“However, this is expected to be a few years in the future before it is mature enough [20].”
Metadata
- Authors
- Pinson, M. H.
- Affiliations
- Institute for Telecommunication Sciences (US)
- Corresponding author
- Pinson, Margaret H.
- ORCID
- Pinson, Margaret H.
- Venue
- IEEE Transactions on Broadcasting
- Published
- 2023-03
- Volume
- 69
- Issue
- 1
- Pages
- 97-117
- ISSN
- 0018-9316 (ISSN-L)
- 1557-9611
- OpenAlex
- W4287888462
- Cited by
- 19
- References
- 95
- Citations by year
- 2025: 8 · 2024: 3 · 2023: 6
- FWCI
- 1.55
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Broadcasting (publishedVersion, cc-by)
- Funders
- Public Safety Communications Research (PSCR) Division of the National Institute of Standards and Technology
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- no-reference quality metrics, image quality assessment, video quality assessment, metric accuracy, reproducibility, Pearson correlation, root cause analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Why no-reference metrics for image and video quality lack accuracy and reproducibility. https://app.cochat.ai/papers/10.1109/tbc.2022.3191059
- The paper
- Pinson, M. H. (2023). Why no-reference metrics for image and video quality lack accuracy and reproducibility. IEEE Transactions on Broadcasting. https://doi.org/10.1109/tbc.2022.3191059
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.