Waterloo Exploration Database: New Challenges for Image Quality Assessment Models
The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
The great content diversity of real-world digital images poses a grand challenge to image quality assessment (IQA) models, which are traditionally designed and validated on a handful of commonly used IQA databases with very limited content variation. To test the generalization capability and to facilitate the wide usage of IQA techniques in real-world applications, we establish a large-scale database named the Waterloo Exploration Database, which in its current state contains 4744 pristine natural images and 94 880 distorted images created from them. Instead of collecting the mean opinion score for each image via subjective testing, which is extremely difficult if not impossible, we present three alternative test criteria to evaluate the performance of IQA models, namely, the pristine/distorted image discriminability test, the listwise ranking consistency test, and the pairwise preference consistency test (P-test). We compare 20 well-known IQA models using the proposed criteria, which not only provide a stronger test in a more challenging testing environment for existing models, but also demonstrate the additional benefits of using the proposed database. For example, in the P-test, even for the best performing no-reference IQA model, more than 6 million failure cases against the model are "discovered" automatically out of over 1 billion test pairs. Furthermore, we discuss how the new database may be exploited using innovative approaches in the future, to reveal the weaknesses of existing IQA models, to provide insights on how to improve the models, and to shed light on how the next-generation IQA models may be developed. The database and codes are made publicly available at: https://ece.uwaterloo.ca/~k29ma/exploration/.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Self-citationMaking a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766
Multiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253
Self-citationNo-Reference Image Quality Assessment in the Spatial Domain
Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2214050 · Cited by 6,064
FSIM: A Feature Similarity Index for Image Quality Assessment
Zhang, L., Zhang, L., Mou, X., & Zhang, D. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2109730 · Cited by 5,203
Self-citationImage information and visual quality
Sheikh, H. R., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.859378 · Cited by 4,104
Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures
Wang, Z., & Bovik, A. C. (2009) · IEEE Signal Processing Magazine · doi:10.1109/msp.2008.930649 · Cited by 3,175
Self-citationA Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms
Sheikh, H. R., Sabir, M. F., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2006.881959 · Cited by 2,966
Most apparent distortion: full-reference image quality assessment and the role of strategy
Larson, E. C., & Chandler, D. M. (2010) · Journal of Electronic Imaging · doi:10.1117/1.3267105 · Cited by 1,953
Blind Image Quality Assessment: From Natural Scene Statistics to Perceptual Quality
Moorthy, A. K., & Bovik, A. C. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2147325 · Cited by 1,763
Blind Image Quality Assessment: A Natural Scene Statistics Approach in the DCT Domain
Saad, M. A., Bovik, A. C., & Charrier, C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2191563 · Cited by 1,751
Gradient Magnitude Similarity Deviation: A Highly Efficient Perceptual Image Quality Index
Xue, W., Zhang, L., Mou, X., & Bovik, A. C. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2293423 · Cited by 1,697
Self-citationA Feature-Enriched Completely Blind Image Quality Evaluator
Zhang, L., Zhang, L., & Bovik, A. C. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2426416 · Cited by 1,311
Self-citationThe Method of Paired Comparisons.
Curnow, R. N., & David, H. A. (1964) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344014 · Cited by 1,271
Image database TID2013: Peculiarities, results and perspectives
Ponomarenko, N., Jin, L., Ieremeiev, O., Lukin, V., Egiazarian, K., Astola, J., Vozel, B., Chehdi, K., Carli, M., Battisti, F., et al. (2015) · Signal Processing: Image Communication · doi:10.1016/j.image.2014.10.009 · Cited by 1,241
A Two-Step Framework for Constructing Blind Image Quality Indices
Moorthy, A. K., & Bovik, A. C. (2010) · IEEE Signal Processing Letters · doi:10.1109/lsp.2010.2043888 · Cited by 1,196
VSNR: A Wavelet-Based Visual Signal-to-Noise Ratio for Natural Images
Chandler, D. M., & Hemami, S. S. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2007.901820 · Cited by 1,147
TID2008 – A database for evaluation of full-reference visual quality assessment metrics
Ponomarenko, N. N., Лукин, В. В., & Zelensky, A. A. (2004) · Iris (Roma Tre University) · Cited by 992
Modern Image Quality Assessment
Wang, Z., & Bovik, A. C. (2006) · Synthesis Lectures on Image, Video, and Multimedia Processing · doi:10.1007/978-3-031-02238-8 · Cited by 967
Self-citationThe visible differences predictor: an algorithm for the assessment of image fidelity
Daly, S. J. (1993) · The MIT Press eBooks · Cited by 828
Unsupervised feature learning framework for no-reference image quality assessment
Ye, P., Kumar, J., Kang, L., & Doermann, D. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247789 · Cited by 785
Massive Online Crowdsourced Study of Subjective and Objective Picture Quality
Ghadiyaram, D., & Bovik, A. C. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2500021 · Cited by 782
Visible differences predictor: an algorithm for the assessment of image fidelity
Daly, S. J. (1992) · SPIE Proceedings · doi:10.1117/12.135952 · Cited by 718
Blind Image Quality Assessment Using Joint Statistics of Gradient Magnitude and Laplacian Features
Xue, W., Mou, X., Zhang, L., Bovik, A. C., & Feng, X. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2355716 · Cited by 641
Using Free Energy Principle For Blind Image Quality Assessment
Gu, K., Zhai, G., Yang, X., & Zhang, W. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2014.2373812 · Cited by 611
Perceptual blur and ringing metrics: application to JPEG2000
Marziliano, P., Dufaux, F., Winkler, S., & Ebrahimi, T. (2004) · Signal Processing: Image Communication · doi:10.1016/j.image.2003.08.003 · Cited by 491
Reduced-reference image quality assessment using a wavelet-domain natural image statistic model
Wang, Z., & Simoncelli, E. P. (2005) · SPIE Proceedings · doi:10.1117/12.597306 · Cited by 438
Self-citationLearning without Human Scores for Blind Image Quality Assessment
Xue, W., Zhang, L., & Mou, X. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.133 · Cited by 404
Self-citationRRED Indices: Reduced Reference Entropic Differencing for Image Quality Assessment
Soundararajan, R., & Bovik, A. C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2166082 · Cited by 337
Objective quality assessment of multiply distorted images
Jayaraman, D., Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012) · 2012 Conference Record of the Forty Sixth Asilomar Conference on Signals, Systems and Computers (ASILOMAR) · doi:10.1109/acssc.2012.6489321 · Cited by 322
Analysis of Public Image and Video Databases for Quality Assessment
Winkler, S. (2012) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2012.2215007 · Cited by 299
Wang, Z., Wu, G., Sheikh, H. R., Simoncelli, E. P., Yang, E.-H., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.864165 · Cited by 293
Self-citationBest Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing
Hossfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., & Tran-Gia, P. (2014) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2291663 · Cited by 283
Subjective quality assessment IRCCyN/IVC database
Callet, P. L., & Autrusseau, F. (2005) · HAL · Cited by 261
A highly efficient method for blind image quality assessment
Wu, Q., Wang, Z., & Li, H. (2015) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2015.7350816 · Cited by 169
Self-citationBlind Image Quality Assessment Based on Multichannel Feature Fusion and Label Transfer
Wu, Q., Li, H., Meng, F., Ngan, K. N., Luo, B., Huang, C., & Zeng, B. (2016) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2412773 · Cited by 164
Self-citationLearning to Rank for Blind Image Quality Assessment
Gao, F., Tao, D., Gao, X., & Li, X. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2377181 · Cited by 146
Wang, Z., & Simoncelli, E. P. (2008) · Journal of Vision · doi:10.1167/8.12.8 · Cited by 138
Self-citationThe method of paired comparisons
David, H. A. 1. (1988) · Cited by 118
Reduced-reference metric design for objective perceptual quality assessment in wireless imaging
Engelke, U., Kusuma, T. M., Zepernick, H., & Caldera, M. (2009) · Signal Processing Image Communication · doi:10.1016/j.image.2009.06.005 · Cited by 95
Biologically inspired image quality assessment
Gao, F., & Yu, J. (2015) · Signal Processing · doi:10.1016/j.sigpro.2015.08.012 · Cited by 68
Utilizing Image Scales Towards Totally Training Free Blind Image Quality Assessment
Saha, A., & Wu, Q. M. J. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2411436 · Cited by 54
Self-citationRank learning on training set selection and image quality assessment
Xu, L., Lin, W., Li, J., Wang, X., Yan, Y., & Fang, Y. (2014) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2014.6890291 · Cited by 8
Research works printed but not linked: 5
- ponomarenko, 2008, Tampere Image Database TID2008
- winkler, 2016, Image and video quality resources
- sheikh, 2016, Image and Video Quality Assessment Research at LIVE
- 2000, Final report from the video quality experts group on the validation of objective models of video quality assessment
- horita, 2010, Toyama-MICT Image Quality Evaluation Database
Cited by
FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising
Zhang, K., Zuo, W., & Zhang, L. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2839891 · Cited by 2,678
Self-citationLearning Deep CNN Denoiser Prior for Image Restoration
Zhang, K., Zuo, W., Gu, S., & Zhang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.300 · Cited by 2,116
“Hence, instead of training on a small dataset consisting of 400 Berkeley segmentation dataset (BSD) images of size 180×180 [13], we collect a large dataset which includes 400 BSD images, 400 selected images from validation set of ImageNet database [20] and 4,744 images of Waterloo Exploration Database [40].”
MethodSelf-citationToward Convolutional Blind Denoising of Real Photographs
Guo, S., Yan, Z., Zhang, K., Zuo, W., & Zhang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00181 · Cited by 1,221
Self-citationLearning a Single Convolutional Super-Resolution Network for Multiple Degradations
Zhang, K., Zuo, W., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00344 · Cited by 1,124
Self-citationPlug-and-Play Image Restoration With Deep Denoiser Prior
Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., & Timofte, R. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3088914 · Cited by 1,113
Self-citation
Metadata
- Authors
- Ma, K., Duanmu, Z., Wu, Q., Wang, Z., Yong, H., Li, H., & Zhang, L.
- Affiliations
- University of Waterloo (CA)
- University of Electronic Science and Technology of China (CN)
- Hong Kong Polytechnic University (HK)
- Venue
- IEEE Transactions on Image Processing
- Published
- 2017-02
- Volume
- 26
- Issue
- 2
- Pages
- 1004-1016
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W2556068545
- Cited by
- 789
- References
- 43
- Citations by year
- 2026: 90 · 2025: 123 · 2024: 106 · 2023: 110 · 2022: 70 · 2021: 92 · 2020: 78 · 2019: 69 · 2018: 35 · 2017: 16
- FWCI
- 26.97
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Image Processing (publishedVersion)
- PubMed (publishedVersion)
- PolyU Institutional Research Archive (Hong Kong Polytechnic University) (submittedVersion)
- Funders
- National Natural Science Foundation of China
- China Scholarship Council
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- image quality assessment, no-reference image quality assessment, mean opinion score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.