Why no-reference metrics for image and video quality lack accuracy and reproducibility
Open accessThe DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index.
Abstract
This article provides a comprehensive overview of no reference (NR) metrics for image quality analysis (IQA) and video quality analysis (VQA). We examine 26 independent evaluations of NR metrics (previously published) and analyze 32 NR metrics on six IQA datasets and six VQA datasets (new results). Where NR metric developers claim Pearson correlation values between 0.66 and 0.99, our measurements range from 0.0 to 0.63. None of the NR metrics we analyzed are accurate enough to be deployed by industry. Performance evaluations that indicate otherwise are based on insufficient data and highly inaccurate. We will examine development strategies, tools, datasets, root cause analysis, and our baseline metric for collaboration, Sawatch.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Similar limitations have been widely discussed in both full-reference and no-reference quality assessment, particularly when objective scores lack reproducibility or fail to generalize across content and distortion conditions [54].”
From A Survey and Tutorial on Image Quality Assessment with a Contrast-Weighted Structural Similarity Framework (2026) · cites this paper as [54]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Traditional full-reference metrics often rely on pixel-wise or structural differences that do not always align with human visual perception, particularly when images contain multiple distortion types, local contrast changes, or domain-specific degradation patterns.”
From A Survey and Tutorial on Image Quality Assessment with a Contrast-Weighted Structural Similarity Framework (2026) · cites this paper as [54]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398 · cited 2×
“LBP [29] was intended for texture classifi-cation.”
BackgroundMaking a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766 · cited 2×
“Retraining is particularly important for ML-NSS metrics, and MATLAB offers tools to re-train NIQE [34] and BRISQUE [10].”
MethodNo-Reference Image Quality Assessment in the Spatial Domain
Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2214050 · Cited by 6,064 · cited 2×
“BRISQUE [10] uses mean subtracted contrast normalized (MSCN) coefficients.”
MethodA Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms
Sheikh, H. R., Sabir, M. F., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.881959 · Cited by 2,966 · cited 3×
“Datasets with conventional experiment designs, like LIVE-2006 [70], avoid media with camera impairments.”
BackgroundMethodInfluentialBlind Image Quality Assessment: From Natural Scene Statistics to Perceptual Quality
Moorthy, A. K., & Bovik, A. C. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2147325 · Cited by 1,763
Blind Image Quality Assessment: A Natural Scene Statistics Approach in the DCT Domain
Saad, M. A., Bovik, A. C., & Charrier, C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2191563 · Cited by 1,751
A new standardized method for objectively measuring video quality
Pinson, M. H., & Wolf, S. (2004) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2004.834028 · Cited by 1,182 · cited 2×
“The best known of those are Video Quality Metric ( VQM ) [102] from ITU-T Rec.”
BackgroundSelf-citationTalebi, H., & Milanfar, P. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2831899 · Cited by 1,023 · cited 1×
“The NR metric with the best accuracy is NIMA [33] with ρ = 0 .”
BackgroundBlind image quality evaluation using perception based features
Venkatanath, N., Praneeth, D., Bh, M. C., Channappayya, S. S., & Medasani, S. S. (2015) · 2015 Twenty First National Conference on Communications (NCC) · doi:10.1109/ncc.2015.7084843 · Cited by 978 · cited 1×
“PIQE [41] takes inspiration from NIQE and BRISQUE , using both circularly-symmetric Gaussian weighting function and MSCN. SpEED-NR [44] uses a Gaussian scale mixture (GSM) model.”
MethodAVA: A large-scale database for aesthetic visual analysis
Murray, N., Marchesotti, L., & Perronnin, F. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247954 · Cited by 936
Massive Online Crowdsourced Study of Subjective and Objective Picture Quality
Ghadiyaram, D., & Bovik, A. C. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2500021 · Cited by 782
KonIQ-10k: An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment
Hosu, V., Lin, H., Sziranyi, T., & Saupe, D. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2967829 · Cited by 759
A no-reference objective image sharpness metric based on the notion of just noticeable blur (JNB)
Ferzli, R., & Karam, L. J. (2009) · IEEE Transactions on Image Processing · doi:10.1109/tip.2008.2011760 · Cited by 758 · cited 1×
“JNB [21] relies upon heuristics obtained from a subjective test that characterizes the response of the human visual system to blurriness.”
MethodNo-reference image quality assessment based on spatial and spectral entropies
Liu, L., Liu, B., Huang, H., & Bovik, A. C. (2014) · Signal Processing: Image Communication · doi:10.1016/j.image.2014.06.006 · Cited by 721
Blind Image Quality Assessment Using Joint Statistics of Gradient Magnitude and Laplacian Features
Xue, W., Mou, X., Zhang, L., Bovik, A. C., & Feng, X. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2355716 · Cited by 641 · cited 1×
“Log-BIQA [30] uses Gradient Magnitude and Laplacian of Gaussian (LOG).”
MethodNo-reference quality assessment using natural scene statistics: JPEG2000
Sheikh, H. R., Bovik, A. C., & Cormack, L. (2005) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.854492 · Cited by 575 · cited 1×
“ADMD [7] and JP2KNR [22] use wavelets.”
KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment
Hosu, V, Lin, H., Szirányi, T., & Saupe, D. (2020) · SZTAKI Publication Repository (Hungarian Academy of Sciences) · Cited by 512
Blind Prediction of Natural Video Quality
Saad, M. A., Bovik, A. C., & Charrier, C. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2299154 · Cited by 471
Learning without Human Scores for Blind Image Quality Assessment
Xue, W., Zhang, L., & Mou, X. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.133 · Cited by 404
UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content
Tu, Z., Wang, Y., Birkbeck, N., Adsumilli, B., & Bovik, A. C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3072221 · Cited by 363 · cited 1×
“Similarly, [48] uses three datasets ( KoNViD-1K , LIVE-Qualcomm, and CVD2014 ) and [98] uses three datasets ( KoNViD-1K , LIVE-VQC and YouTube-UGC ), which they refer to as UGC-VQA .”
MethodA Completely Blind Video Integrity Oracle
Mittal, A., Saad, M. A., & Bovik, A. C. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2502725 · Cited by 357
Two-Level Approach for No-Reference Consumer Video Quality Assessment
Korhonen, J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2923051 · Cited by 354 · cited 1×
“Similarly, [48] uses three datasets ( KoNViD-1K , LIVE-Qualcomm, and CVD2014 ) and [98] uses three datasets ( KoNViD-1K , LIVE-VQC and YouTube-UGC ), which they refer to as UGC-VQA .”
MethoddipIQ: Blind Image Quality Assessment by Learning-to-Rank Discriminable Image Pairs
Ma, K., Liu, W., Liu, T., Wang, Z., & Tao, D. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2708503 · Cited by 353 · cited 3×
“Metric dipIQ [16] is trained on data from a FR metric, which was fed into a pairwise learning to rank (L2R) algorithm.”
MethodInfluentialThe Konstanz natural video database (KoNViD-1k)
Hosu, V., Hahn, F., Jenadeleh, M., Lin, H., Men, H., Sziranyi, T., Li, S., & Saupe, D. (2017) · 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2017.7965673 · Cited by 343
No-Reference Quality Assessment of Contrast-Distorted Images Based on Natural Scene Statistics
Fang, Y., Ma, K., Wang, Z., Lin, W., Fang, Z., & Zhai, G. (2014) · IEEE Signal Processing Letters · doi:10.1109/lsp.2014.2372333 · Cited by 339 · cited 2×
“We will refer to these as machine learning NSS ( ML-NSS ) metrics, to avoid confusion with the NSS metric [39].”
MethodPerceptual quality prediction on authentically distorted images using a bag of features approach
Ghadiyaram, D., & Bovik, A. C. (2017) · Journal of Vision · doi:10.1167/17.1.32 · Cited by 322
Large-Scale Study of Perceptual Video Quality
Sinno, Z., & Bovik, A. C. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2869673 · Cited by 310 · cited 1×
“Similar conclusions appear in [72] and [73].”
Blind image quality assessment by relative gradient statistics and adaboosting neural network
Liu, L., Hua, Y., Zhao, Q., Huang, H., & Bovik, A. C. (2016) · Signal Processing: Image Communication · doi:10.1016/j.image.2015.10.005 · Cited by 282 · cited 1×
“OG-IQA [40] uses the gradient orientation and magnitude.”
MethodYouTube UGC Dataset for Video Compression Research
Wang, Y., Inguva, S., & Adsumilli, B. (2019) · 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2019.8901772 · Cited by 282 · cited 1×
“YouTube-UGC [82], [83] by Google R (cid:6) provides an independent industry assessment of NR metrics for the UGC use case.”
MethodNo-Reference Blur Assessment of Digital Pictures Based on Multifeature Classifiers
Ciancio, A., da Costa, A. L. N. T., da Silva, E. A. B., Said, A., Samadani, R., & Obrador, P. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2053549 · Cited by 251
Gu, K., Wang, S., Zhai, G., Ma, S., Yang, X., Lin, W., Zhang, W., & Gao, W. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2518868 · Cited by 244 · cited 1×
“BTMQI [11] detects tone-mapped images (i.e., converted from high dynamic range to low dynamic range).”
BackgroundCVD2014—A Database for Evaluating No-Reference Video Quality Assessment Algorithms
Nuutinen, M., Virtanen, T., Vaahteranoksa, M., Vuori, T., Oittinen, P., & Hakkinen, J. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2562513 · Cited by 227
A no-reference perceptual image sharpness metric based on a cumulative probability of blur detection
Narvekar, N. D., & Karam, L. J. (2009) · 2009 International Workshop on Quality of Multimedia Experience · doi:10.1109/qomex.2009.5246972 · Cited by 208 · cited 1×
“CPBD [13] models human perception of localized blur.”
BackgroundNo-reference image quality assessment in curvelet domain
Liu, L., Dong, H., Huang, H., & Bovik, A. C. (2014) · Signal Processing: Image Communication · doi:10.1016/j.image.2014.02.004 · Cited by 177
CID2013: A Database for Evaluating No-Reference Image Quality Assessment Algorithms
Virtanen, T., Nuutinen, M., Vaahteranoksa, M., Oittinen, P., & Hakkinen, J. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2378061 · Cited by 176
In-Capture Mobile Video Distortions: A Study of Subjective Behavior and Objective Algorithms
Ghadiyaram, D., Pan, J., Bovik, A. C., Moorthy, A. K., Panda, P., & Yang, K.-C. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2707479 · Cited by 154 · cited 1×
“Similar conclusions appear in [72] and [73].”
SpEED-QA: Spatial Efficient Entropic Differencing for Image and Video Quality
Bampis, C. G., Gupta, P., Soundararajan, R., & Bovik, A. C. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2726542 · Cited by 150 · cited 3×
“The first step, an NR metric that we refer to as 2stepQA-NR, is an NR constrained variant of SpEED-QA [44].”
MethodInfluentialZhang, Y., Moorthy, A. K., Chandler, D. M., & Bovik, A. C. (2014) · Signal Processing: Image Communication · doi:10.1016/j.image.2014.05.004 · Cited by 102
Large-Scale Crowdsourced Study for Tone-Mapped HDR Pictures
Kundu, D., Ghadiyaram, D., Bovik, A. C., & Evans, B. L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2713945 · Cited by 102
The Influence of Subjects and Environment on Audiovisual Subjective Tests: An International Study
Pinson, M., Janowski, L., Pépion, R., Huynh‐Thu, Q., Schmidmer, C., Corriveau, P., Younkin, A. C., Callet, P. L., Barkowsky, M., & Ingram, W. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2215306 · Cited by 96 · cited 2×
“olds are informed by analyses of subject ratings in [77], [87], and [88].”
BackgroundSelf-citationSun, W., Gu, K., Ma, S., Zhu, W., Liu, N., & Zhai, G. (2018) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2018.8547102 · Cited by 94
The Accuracy of Subjects in a Quality Experiment: A Theoretical Subject Model
Janowski, L., & Pinson, M. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2484963 · Cited by 83 · cited 3×
“A study of subject rating behaviors [87] shows that subjects’ scoring is a random process.”
BackgroundInfluentialSelf-citationStudy of 3D Virtual Reality Picture Quality
Chen, M., Jin, Y., Goodall, T., Yu, X., & Bovik, A. C. (2019) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2019.2956408 · Cited by 83
Blind Quality Assessment of Camera Images Based on Low-Level and High-Level Statistical Features
Liu, Y., Gu, K., Wang, S., Zhao, D., & Gao, W. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2849602 · Cited by 77
KonVid-150k: A Dataset for No-Reference Video Quality Assessment of Videos in-the-Wild
Götz-Hahn, F., Hosu, V., Lin, H., & Saupe, D. (2021) · IEEE Access · doi:10.1109/access.2021.3077642 · Cited by 68
Subjective and Objective Quality Assessment of High Frame Rate Videos
Madhusudana, P. C., Yu, X., Birkbeck, N., Wang, Y., Adsumilli, B., & Bovik, A. C. (2021) · IEEE Access · doi:10.1109/access.2021.3100462 · Cited by 65
Modeling subjective tests of quality of experience with a Generalized Linear Model
Janowski, L., & Papir, Z. (2009) · 2009 International Workshop on Quality of Multimedia Experience · doi:10.1109/qomex.2009.5246979 · Cited by 62
Perceptual Quality Assessment of Internet Videos
Xu, J., Li, J., Zhou, X., Zhou, W., Wang, B., & Chen, Z. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475486 · Cited by 58
NoRM: No‐Reference Image Quality Metric for Realistic Image Synthesis
Herzog, R., Čadík, M., Aydčin, T. O., Kim, K. I., Myszkowski, K., & Seidel, H. (2012) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2012.03055.x · Cited by 56 · cited 1×
“NoRM [36] detects 3D rendering artifacts.”
Predicting the Quality of Images Compressed After Distortion in Two Steps
Yu, X., Bampis, C. G., Gupta, P., & Bovik, A. C. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2922850 · Cited by 55 · cited 2×
“2stepQA [6] is a two-step reduced reference (RR) metric.”
BackgroundMethodPerceptual quality assessment for H.264/AVC compression
Romaniak, P., Janowski, L., Leszczuk, M., & Papir, Z. (2012) · IEEE Consumer Communications and Networking Conference · doi:10.1109/ccnc.2012.6181021 · Cited by 51 · cited 1×
“The Key Indicators [23]–[28] are a set of 15 RCA metrics that detect blackout (all picture content lost), blockiness, block loss, blur, contrast, exposure, flickering, freezing, interlacing, letter-boxing, noise, pillar-boxing, slicing, spatial activity, and temporal activity.”
BackgroundConstrained sampling experiments reveal principles of detection in natural scenes
Sebastian, S., Abrams, J., & Geisler, W. S. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1619487114 · Cited by 51 · cited 1×
“The second strategy applies the theory of natural scene statistics (NSS) from [95] to identify structural patterns or irregularities in the media that characterize compression or other artifacts.”
BackgroundEncoding Visual Sensitivity by MaxPol Convolution Filters for Image Sharpness Assessment
Hosseini, M. S., Zhang, Y., & Plataniotis, K. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2906582 · Cited by 51 · cited 14×
“Examples include the likelihood that computer vision (CV) will succeed or fail ( iAITech-NJIT , DIQA dataset) and automatic focusing of digital pathology slides ( HVS-MaxPol ).”
BackgroundInfluentialA Subjective and Objective Study of Space-Time Subsampled Video Quality
Lee, D. Y., Paul, S., Bampis, C. G., Ko, H., Kim, J.-H., Jeong, S., Homan, B., & Bovik, A. C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3137658 · Cited by 47
Selecting scenes for 2D and 3D subjective video quality tests
Pinson, M., Barkowsky, M., & Callet, P. L. (2013) · EURASIP Journal on Image and Video Processing · doi:10.1186/1687-5281-2013-50 · Cited by 46 · cited 1×
“AGH/NTIA/Dolby , CCRIQ, and CCRIQ2 use the systematic selection criteria from [86].”
BackgroundSelf-citationA Comparative Study of Algorithms for Realtime Panoramic Video Blending
Zhu, Z., Lu, J., Wang, M., Zhang, S., Martin, R. R., Liu, H., & Hu, S. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2808766 · Cited by 45
Recent developments in visual quality monitoring by key performance indicators
Leszczuk, M., Hanusiak, M., Farias, M. C. Q., Wyckens, E., & Heston, G. (2014) · Multimedia Tools and Applications · doi:10.1007/s11042-014-2229-2 · Cited by 43
A no-reference objective image quality metric based on perceptually weighted local noise
Zhu, T., & Karam, L. J. (2014) · EURASIP Journal on Image and Video Processing · doi:10.1186/1687-5281-2014-5 · Cited by 41 · cited 1×
“NR-PWN [38] applies a perceptual noisiness model.”
Panetta, K. A., Samani, A., & Agaian, S. С. (2018) · IEEE Access · doi:10.1109/access.2018.2804901 · Cited by 32 · cited 4×
“See Figure 1 to visualize the triangle patterns: TDMEC is an upper triangle shape, and dip-IQ is a lower triangle shape.”
MethodInfluentialRecent advances in multimedia networking
Cerqueira, E., Zeadally, S., Leszczuk, M., Curado, M., & Mauthe, A. (2010) · Multimedia Tools and Applications · doi:10.1007/s11042-010-0578-z · Cited by 29
Perceptual evaluation of single-image super-resolution reconstruction
Wang, G., Li, L., Li, Q., Gu, K., Lu, Z., & Qian, J. (2017) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2017.8296862 · Cited by 29
Subjective Quality Assessment For Youtube Ugc Dataset
Yim, J. G., Wang, Y., Birkbeck, N., & Adsumilli, B. (2020) · Proceedings - International Conference on Image Processing · doi:10.1109/icip40778.2020.9191194 · Cited by 29
Application-Driven No-Reference Quality Assessment for Dermoscopy Images With Multiple Distortions
Xie, F., Lu, Y., Bovik, A. C., Jiang, Z., & Meng, R. (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2015.2493580 · Cited by 28 · cited 3×
“ADMD [7] and JP2KNR [22] use wavelets.”
BackgroundImpact of camera pixel count and monitor resolution perceptual image quality
Saad, M. A., Pinson, M., Nicholas, D., Kets, N. V., Wallendael, G. V., Silva, R. D., Jaladi, R., & Corriveau, P. J. (2015) · Colour and Visual Computing Symposium (CVCS) · doi:10.1109/cvcs.2015.7274887 · Cited by 24 · cited 3×
“This difference is ≈ 0.2 MOS for high quality images and ≈ 0.0 MOS for low quality images [56].”
BackgroundMethodInfluentialSelf-citationVideo Quality Model for Variable Frame Delay (VQM_VFD)
Wolf, S., & Pinson, M. (2011) · doi:10.70220/4fbgs8f1 · Cited by 23 · cited 2×
“BT.1683 (2004) and Video Quality Metric for Variable Frame Delay ( VQM-VFD ) [103].”
BackgroundMethodSelf-citationKatsigiannis, S., Scovell, J. J., Ramzan, N., Janowski, L., Corriveau, P. J., Saad, M. A., & Wallendael, G. V. (2018) · Quality and User Experience · doi:10.1007/s41233-018-0019-8 · Cited by 23 · cited 2×
“Some NR metrics, like VIQET [50]–[51] and Sawatch [3], deploy multiple strategies and combine the outputs of multiple NR metrics.”
BackgroundMethodOuttas, M., Zhang, L., Déforges, O., Hammidouche, W., Serir, A., & Cavaro‐Ménard, C. (2016) · International Symposium on Signal, Image, Video and Communications (ISIVC) · doi:10.1109/isivc.2016.7894006 · Cited by 22 · cited 2×
“NIQE-K [35] models the opinion of radiologists when viewing ultrasound images.”
MethodPerceptual Quality Assessment of UHD-HDR-WCG Videos
Athar, S., Costa, T., Zeng, K., & Wang, Z. (2019) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2019.8803179 · Cited by 20
A Multiple Attributes Image Quality Database for Smartphone Camera Photo Quality Assessment
Zhu, W., Zhai, G.-T., Han, Z.-X., Min, X.-K., Wang, T., Zhang, Z.-C., & Yangand, X. (2020) · International Conference on Information Photonics · doi:10.1109/icip40778.2020.9191104 · Cited by 20
Document Image Quality Assessment Using Discriminative Sparse Representation
Peng, X., Cao, H., & Natarajan, P. (2016) · IAPR International Workshop on Document Analysis Systems (DAS) · doi:10.1109/das.2016.24 · Cited by 18 · cited 1×
“The DIQA [61] dataset contains scanned documents and simulated “ratings” that assess the likelihood that optical character recognition (OCR) will succeed, by comparing the original document with the text produced by OCR.”
BackgroundLeszczuk, M. (2011) · Communications in computer and information science · doi:10.1007/978-3-642-21512-4_11 · Cited by 17
Image Sharpness Metric Based on Maxpol Convolution Kernels
Hosseini, M. S., & Plataniotis, K. (2018) · International Conference on Information Photonics · doi:10.1109/icip.2018.8451488 · Cited by 17 · cited 1×
“MaxPol [31] and HVS-MaxPol [19] model the relative sensitivity of the human visual system to image blur, using a convolutional filter.”
MethodSubjective Assessment Experiments That Recruit Few Observers With Repetitions (FOWR)
Pérez, P., Janowski, L., Garcia, N. N., & Pinson, M. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3098450 · Cited by 17 · cited 1×
“However, [93] challenges the concept of anchor conditions or a “common set” in cross-lab experiments.”
BackgroundSelf-citationITS4S: A Video Quality Dataset with Four-Second Unrepeated Scenes
Pinson, M. (2018) · doi:10.70220/2mpf0j1h · Cited by 16 · cited 3×
“We can infer a complex relationship between MOS, quality, and impairments by examining Table X, which shows the relationship between the ITS4S dataset [32] and RCA metrics.”
BackgroundMethodInfluentialSelf-citationTransform domain measure of enhancement — TDME — For security imaging applications
Samani, A., Panetta, K. A., & Agaian, S. С. (2013) · IEEE Conference on Technologies for Homeland Security (HST) · doi:10.1109/ths.2013.6699012 · Cited by 15 · cited 1×
“TDME [46] and TDMEC [47] use a discrete cosine transform (DCT) to detect contrast enhancement.”
MethodAhmed, I. T., Der, C. S., Jamil, N., & Hammad, B. T. (2019) · IEEE Control and System Graduate Research Colloquium (ICSGRC) · doi:10.1109/icsgrc.2019.8837095 · Cited by 15 · cited 1×
“NSS [39] uses the five statistics from NR-IQA-CDI [37].”
MethodITS4S2: An Image Quality Dataset With Unrepeated Images From Consumer Cameras
Pinson, M. (2019) · doi:10.70220/478jr91r · Cited by 13
Self-citationPinson, M. (2020) · doi:10.70220/kk0d38dw · Cited by 11 · cited 3×
“A solution is proposed in [88], where statistics gathered from 60 subjective tests and 90 labto-lab comparisons are used to conclude whether an NR”
BackgroundSelf-citationEvaluating experiment design with unrepeated scenes for video quality subjective assessment
Janowski, L., Malfait, L., & Pinson, M. (2019) · Quality and User Experience · doi:10.1007/s41233-019-0026-4 · Cited by 10 · cited 1×
“…camera impairments • Portray all state-of-the-art camera applications • Assess various display devices • Implement an unrepeated scene design [94] • Exclude outdated impairments • Exclude temporal integration • Exclude transmission errors • Contain images or videos of 4 s duration This…”
BackgroundSelf-citationA Metric for Video Blending Quality Assessment
Zhu, Z., Liu, H., Lu, J., & Hu, S. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2955294 · Cited by 9
Lim, C.-L., & Paramesran, R. (2014) · International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS) · doi:10.1109/ispacs.2014.7024421 · Cited by 8
SIQD: Surveillance Image Quality Database and Performance Evaluation for Objective Algorithms
Zhu, W., Zhai, G., Yao, C., & Yang, X. (2018) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip.2018.8698737 · Cited by 8
Garcia, M. N., & Raake, A. (2010) · International Workshop on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2010.5517960 · Cited by 7
Pinson, M. (2019) · doi:10.70220/3rlnl9jw · Cited by 7 · cited 1×
“The ITS4S3 dataset [65] depicts subject matter used by first responders: crime scenes, fireground, prison riots, search and rescue, and cityscapes.”
Self-citationStudy of Subjective Data Integrity for Image Quality Data Sets with Consumer Camera Content
Nawała, J., Pinson, M., Leszczuk, M., & Janowski, L. (2020) · Journal of Imaging · doi:10.3390/jimaging6030007 · Cited by 7 · cited 1×
“Convenience sampling uses media conveniently available, which produces biased results (e.g., see the analysis of VIME1 in [58]).”
BackgroundSelf-citationCompensating for Type-I errors in video quality assessment
Brunnstrom, K. E., Tavakoli, S., & Søgaard, J. (2015) · International Workshop on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2015.7148080 · Cited by 6 · cited 1×
“See [89] for details.”
Technology Gaps in First Responder Cameras
Pinson, M. (2017) · doi:10.70220/l83fl1hk · Cited by 6 · cited 1×
“Task specific concerns impact how first responders describe the quality of media [90] and by consequence may subtly impact MOSs.”
BackgroundSelf-citationEnhancing Temporal Quality Measurements in a Globally Deployed Streaming Video Quality Predictor
Bampis, C. G., Li, Z., & Bovik, A. C. (2018) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2018.8451275 · Cited by 5 · cited 1×
“The large number of datasets and RCA / HVS strategy produced metrics that are resilient to advances in video technology, as demonstrated by [104].”
MethodAn Innovative Video Quality Assessment Method and An Impairment Video Dataset
Shi, H., & Liu, C. (2021) · IEEE International Conference on Imaging Systems and Techniques (IST) · doi:10.1109/ist50367.2021.9651418 · Cited by 5 · cited 1×
“The developers of the iAITech-NJIT NR metrics note differences between human perception and their CV use case [20].”
BackgroundNo-reference image quality assessment for removal of processed and unprocessed noise
Rakhshanfar, M., & Amer, M. A. (2015) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2015.7351187 · Cited by 4 · cited 1×
“Other SSP metrics include AGWN [8], Entropy Noise [17], and LBP [29].”
BackgroundITS4S4: A Video Quality Study of Camera Pans
Pinson, M., & Elting, S. (2019) · doi:10.70220/740j2cvp · Cited by 4
Self-citationAnalysis of No-Reference Metrics for Image and Video Quality of Consumer Applications
Pinson, M. (2020) · doi:10.70220/hb2drbf6 · Cited by 3 · cited 3×
“The vqegHDcuts dataset uses such a method to merge multiple VQEG datasets [84].”
MethodInfluentialSelf-citationRevealing the dark side of a subjective study: Learnings from noise and sharpness ratings
Saad, M. A., Corriveau, P. J., & Jaladi, R. (2015) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2015.7148121 · Cited by 2 · cited 2×
“Intel [1] evaluated six NR-IQA metrics on consumer content and reported that “those algorithms did not correlate well with human perceptual judgement of image quality.””
BackgroundMethodSubjective Study of Perceptual Quality for Micro-Video Applications
Hu, Y., Zhang, Y., Liu, Z., Chen, Z., & Liu, S. (2020) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr49039.2020.00054 · Cited by 2
Pinson, M., Corriveau, P. J., Leszczuk, M., & Colligan, M. (2020) · Electronic Imaging · doi:10.2352/issn.2470-1173.2020.11.hvei-092 · Cited by 1
Self-citation
Research works printed but not linked: 12
- 2020, National Telecommunications and Information Administration, Institute for Telecommunication Sciences, NR Metric Framework
- Katsavounidis, 2018, Digital Optimizer—A Perceptual Video Encoding Optimization Framework.
- 2021, AGH Video Qualtiy of Experience (QoE Team) Indicators
- 2008, Subjective Video Quality Assessment Methods for Multimedia Applications
- 2021, VQET Image Quality Evaluation Tool (VIQET)
- Kahneman, 2021, Noise: A Flaw in Human Judgement
- 10.1007/s41233-019-0026-4 · DOI
- 2021, Laboratory for Image & Video Engineering
- 2021, The Konstanz Visual Quality Databases
- 2022, VQEG HDTV Test. Vqeghd1
- Lee, 2020, This Ugly T-Shirt Makes You Invisible to Facial Recognition Tech
- 2022, Sawatch Demo
Cited by
Deep neural network for blind visual quality assessment of 4K content
Lu, W., Sun, W., Min, X., Zhu, W., Zhou, Q., He, J., Wang, Q., Zhang, Z., Wang, T., & Zhai, G. (2023) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2022.3221689 · Cited by 60
Deep learning techniques for atmospheric turbulence removal: a review
Hill, P., Anantrasirichai, N., Achim, A., & Bull, D. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11086-6 · Cited by 43
Super-resolution image visual quality assessment based on structure–texture features
Zhou, F., Wei, S., Lu, Z., Kang, B., Chen, M., & Qiu, G. (2023) · Signal Processing Image Communication · doi:10.1016/j.image.2023.117025 · Cited by 20
Beyond Correlation: Evaluating Multimedia Quality Models With the Constrained Concordance Index
Ragano, A., Martinez, H. B., & Hines, A. (2024) · IEEE transactions on multimedia · doi:10.1109/tmm.2025.3542991 · Cited by 8
“However, with only 6 raters, this interval can expand to more than 1.5 [32].”
BackgroundInfluentialQuality measurement methods for video assisting refereeing systems
Brunnstrom, K. E., Djupsjöbacka, A., Ozoliņš, O., Billingham, J., Wistel, K., & Evans, N. (2023) · Sports Engineering · doi:10.1007/s12283-023-00408-6 · Cited by 8
“However, this is expected to be a few years in the future before it is mature enough [20].”
Metadata
- Authors
- Pinson, M. H.
- Affiliations
- Institute for Telecommunication Sciences (US)
- Corresponding author
- Pinson, Margaret H.
- ORCID
- Pinson, Margaret H.
- Venue
- IEEE Transactions on Broadcasting
- Published
- 2023-03
- Volume
- 69
- Issue
- 1
- Pages
- 97-117
- ISSN
- 0018-9316 (ISSN-L)
- 1557-9611
- OpenAlex
- W4287888462
- Cited by
- 19
- References
- 95
- Citations by year
- 2025: 8 · 2024: 3 · 2023: 6
- FWCI
- 1.55
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Broadcasting (publishedVersion, cc-by)
- Funders
- Public Safety Communications Research (PSCR) Division of the National Institute of Standards and Technology
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Keywords
- no-reference quality metrics, image quality assessment, video quality assessment, metric accuracy, reproducibility, Pearson correlation, root cause analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.