The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms
Abstract
Measurement of visual quality is of fundamental importance for numerous image and video processing applications, where the goal of quality assessment (QA) algorithms is to automatically assess the quality of images or videos in agreement with human quality judgments. Over the years, many researchers have taken different approaches to the problem and have contributed significant research in this area and claim to have made progress in their respective domains. It is important to evaluate the performance of these algorithms in a comparative setting and analyze the strengths and weaknesses of these methods. In this paper, we present results of an extensive subjective quality assessment study in which a total of 779 distorted images were evaluated by about two dozen human subjects. The "ground truth" image quality data obtained from about 25,000 individual human quality judgments is used to evaluate the performance of several prominent full-reference image quality assessment algorithms. To the best of our knowledge, apart from video quality studies conducted by the Video Quality Experts Group, the study presented in this paper is the largest subjective image quality study in the literature in terms of number of images, distortion types, and number of human judgments per image. Moreover, we have made the data from the study freely available to the research community. This would allow other researchers to easily report comparative results in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Multiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253
Self-citationImage quality measures and their performance
Eskicioglu, A. M., & Fisher, P. S. (1995) · IEEE Transactions on Communications · doi:10.1109/26.477498 · Cited by 2,117
Cited by
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Making a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766
Self-citationNo-Reference Image Quality Assessment in the Spatial Domain
Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2214050 · Cited by 6,064
Self-citationNTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study
Agustsson, E., & Timofte, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.150 · Cited by 4,295
Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures
Wang, Z., & Bovik, A. C. (2009) · IEEE Signal Processing Magazine · doi:10.1109/msp.2008.930649 · Cited by 3,175
Self-citationMost apparent distortion: full-reference image quality assessment and the role of strategy
Larson, E. C., & Chandler, D. M. (2010) · Journal of Electronic Imaging · doi:10.1117/1.3267105 · Cited by 1,953
Blind Image Quality Assessment: From Natural Scene Statistics to Perceptual Quality
Moorthy, A. K., & Bovik, A. C. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2147325 · Cited by 1,763
Self-citationInformation content weighting for perceptual image quality assessment
Wang, Z., & Li, Q. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2092435 · Cited by 1,304
A Feature-Enriched Completely Blind Image Quality Evaluator
Zhang, L., Zhang, L., & Bovik, A. C. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2426416 · Cited by 1,311
Self-citationImage database TID2013: Peculiarities, results and perspectives
Ponomarenko, N., Jin, L., Ieremeiev, O., Lukin, V., Egiazarian, K., Astola, J., Vozel, B., Chehdi, K., Carli, M., Battisti, F., et al. (2015) · Signal Processing: Image Communication · doi:10.1016/j.image.2014.10.009 · Cited by 1,241
A Two-Step Framework for Constructing Blind Image Quality Indices
Moorthy, A. K., & Bovik, A. C. (2010) · IEEE Signal Processing Letters · doi:10.1109/lsp.2010.2043888 · Cited by 1,196
Self-citationStudy of Subjective and Objective Quality Assessment of Video
Seshadrinathan, K., Soundararajan, R., Bovik, A. C., & Cormack, L. K. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2010.2042111 · Cited by 1,160
Self-citationVSNR: A Wavelet-Based Visual Signal-to-Noise Ratio for Natural Images
Chandler, D. M., & Hemami, S. S. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2007.901820 · Cited by 1,147
Waterloo Exploration Database: New Challenges for Image Quality Assessment Models
Ma, K., Duanmu, Z., Wu, Q., Wang, Z., Yong, H., Li, H., & Zhang, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2631888 · Cited by 789
Massive Online Crowdsourced Study of Subjective and Objective Picture Quality
Ghadiyaram, D., & Bovik, A. C. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2500021 · Cited by 782
Self-citationKonIQ-10k: An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment
Hosu, V., Lin, H., Sziranyi, T., & Saupe, D. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2967829 · Cited by 759
Motion Tuned Spatio-Temporal Quality Assessment of Natural Videos
Seshadrinathan, K., & Bovik, A. C. (2010) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2034992 · Cited by 708
Self-citationKADID-10k: A Large-scale Artificially Distorted IQA Database
Lin, H., Hosu, V., & Saupe, D. (2019) · 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2019.8743252 · Cited by 523
No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency
Golestaneh, S. A., Dadsetan, S., & Kitani, K. M. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00404 · Cited by 465
From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality
Ying, Z., Niu, H., Gupta, P., Mahajan, D., Ghadiyaram, D., & Bovik, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00363 · Cited by 406
Self-citationA Completely Blind Video Integrity Oracle
Mittal, A., Saad, M. A., & Bovik, A. C. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2502725 · Cited by 357
Self-citationVisual Importance Pooling for Image Quality Assessment
Moorthy, A. K., & Bovik, A. C. (2009) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2009.2015374 · Cited by 340
Self-citationPieAPP: Perceptual Image-Error Assessment Through Pairwise Preference
Prashnani, E., Cai, H., Mostofi, Y., & Sen, P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00194 · Cited by 339
UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content
Tu, Z., Wang, Y., Birkbeck, N., Adsumilli, B., & Bovik, A. C. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3072221 · Cited by 363
“Several authors report SOTA performance on legacy synthetic distortion databases [68], [69] or on naturally distorted data-”
BackgroundSelf-citationBlind image quality assessment through anisotropy
Gabarda, S., & Cristóbal, G. (2007) · Journal of the Optical Society of America A · doi:10.1364/josaa.24.000b42 · Cited by 301
Reduced- and No-Reference Image Quality Assessment
Wang, Z., & Bovik, A. (2011) · IEEE Signal Processing Magazine · doi:10.1109/msp.2011.942471 · Cited by 280
Self-citationKim, J., Zeng, H., Ghadiyaram, D., Lee, S., Zhang, L., & Bovik, A. C. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2736018 · Cited by 283
Self-citationA comprehensive assessment of the structural similarity index
Dosselmann, R., & Yang, X. D. (2011) · Signal, Image and Video Processing · doi:10.1007/s11760-009-0144-1 · Cited by 275
Image Quality Assessment Based on Multiscale Geometric Analysis
Gao, X., Lu, W., Tao, D., & Li, X. (2009) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2018014 · Cited by 252
CVD2014—A Database for Evaluating No-Reference Video Quality Assessment Algorithms
Nuutinen, M., Virtanen, T., Vaahteranoksa, M., Vuori, T., Oittinen, P., & Hakkinen, J. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2562513 · Cited by 227
ViS3: an algorithm for video quality assessment via analysis of spatial and spatiotemporal slices
Vu, P. V., & Chandler, D. M. (2014) · Journal of Electronic Imaging · doi:10.1117/1.jei.23.1.013016 · Cited by 185
Dehazing Evaluation: Real-World Benchmark Datasets, Criteria, and Baselines
Zhao, S., Zhang, L., Huang, S., Shen, Y., & Zhao, S. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2995264 · Cited by 169
Blind omnidirectional image quality assessment with viewport oriented graph convolutional networks
Xu, J., Zhou, W., & Chen, Z. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3015186 · Cited by 166
Considering Temporal Variations of Spatial Visual Distortions in Video Quality Assessment
Ninassi, A., Le Meur, O., Le Callet, P., & Barba, D. (2009) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2009.2014806 · Cited by 159
In-Capture Mobile Video Distortions: A Study of Subjective Behavior and Objective Algorithms
Ghadiyaram, D., Pan, J., Bovik, A. C., Moorthy, A. K., Panda, P., & Yang, K.-C. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2707479 · Cited by 154
Self-citationSpEED-QA: Spatial Efficient Entropic Differencing for Image and Video Quality
Bampis, C. G., Gupta, P., Soundararajan, R., & Bovik, A. C. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2726542 · Cited by 150
Self-citationA Gabor Feature-Based Quality Assessment Model for the Screen Content Images
Ni, Z., Zeng, H., Ma, L., Hou, J., Chen, J., & Ma, K.-K. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2839890 · Cited by 118
Zhang, Y., Moorthy, A. K., Chandler, D. M., & Bovik, A. C. (2014) · Signal Processing: Image Communication · doi:10.1016/j.image.2014.05.004 · Cited by 102
Self-citationDesign of Linear Equalizers Optimized for the Structural Similarity Index
Channappayya, S. S., Bovik, A. C., Caramanis, C., & Heath, R. W. (2008) · IEEE Transactions on Image Processing · doi:10.1109/tip.2008.921328 · Cited by 98
Self-citationAssessing Image Quality Issues for Real-World Problems
Chiu, T.-Y., Zhao, Y., & Gurari, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00370 · Cited by 63
Deep neural network for blind visual quality assessment of 4K content
Lu, W., Sun, W., Min, X., Zhu, W., Zhou, Q., He, J., Wang, Q., Zhang, Z., Wang, T., & Zhai, G. (2023) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2022.3221689 · Cited by 60
No-reference video quality assessment in the compressed domain
Lin, X., Ma, H., Luo, L., & Chen, Y. (2012) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2012.6227454 · Cited by 50
Improving salt and pepper noise removal using a fuzzy mathematical morphology-based filter
González-Hidalgo, M., Massanet, S., Mir, A., & Ruiz-Aguilera, D. (2018) · Applied Soft Computing · doi:10.1016/j.asoc.2017.11.030 · Cited by 49
BBAND INDEX: A NO-REFERENCE BANDING ARTIFACT PREDICTOR
Tu, Z., Lin, J., Wang, Y., Adsumilli, B., & Bovik, A. C. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053634 · Cited by 46
Self-citationOszust, M. (2016) · PLOS ONE · doi:10.1371/journal.pone.0158333 · Cited by 39
Perceptually significant spatial pooling techniques for image quality assessment
Moorthy, A. K., & Bovik, A. C. (2009) · SPIE Proceedings · doi:10.1117/12.810166 · Cited by 37
Self-citationImage quality assessment based on Prewitt magnitude
Zhang, H., Zhu, Q., Fan, C., & Deng, D. (2013) · AEU - International Journal of Electronics and Communications · doi:10.1016/j.aeue.2013.04.001 · Cited by 27
Quality assessment in the era of large models: A survey
Zhang, Z., Zhou, Y., Li, C., Zhao, B., Liu, X., & Zhai, G. (2025) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3722559 · Cited by 23
Why no-reference metrics for image and video quality lack accuracy and reproducibility
Pinson, M. H. (2023) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2022.3191059 · Cited by 19
(2022) · doi:10.3390/electronics11040559 · Cited by 19
Helping Visually Impaired People Take Better Quality Pictures
Mandal, M., Ghadiyaram, D., Gurari, D., & Bovik, A. C. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3282067 · Cited by 14
Self-citationA Combined Full-Reference Image Quality Assessment Method Based on Convolutional Activation Maps
Varga, D. (2020) · Algorithms · doi:10.3390/a13120313 · Cited by 11
No-reference image quality assessment based on global awareness
Hu, Z., Yang, G., Du, Z., Huang, X., Zhang, P., & Liu, D. (2024) · PLOS ONE · doi:10.1371/journal.pone.0310206 · Cited by 5
Metadata
- Authors
- Sheikh, H. R., Sabir, M. F., & Bovik, A. C.
- Venue
- IEEE Transactions on Image Processing
- Published
- 2006-11
- Volume
- 15
- Issue
- 11
- Pages
- 3440-3451
- ISSN
- 1057-7149 (ISSN-L)
- OpenAlex
- W2161907179
- Cited by
- 2,966
- References
- 2
- FWCI
- 51.75
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
