Group MAD Competition? A New Methodology to Compare Objective Image Quality Models
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Objective image quality assessment (IQA) models aim to automatically predict human visual perception of image quality and are of fundamental importance in the field of image processing and computer vision. With an increasing number of IQA models proposed, how to fairly compare their performance becomes a major challenge due to the enormous size of image space and the limited resource for subjective testing. The standard approach in literature is to compute several correlation metrics between subjective mean opinion scores (MOSs) and objective model predictions on several well-known subject-rated databases that contain distorted images generated from a few dozens of source images, which however provide an extremely limited representation of real-world images. Moreover, most IQA models developed on these databases often involve machine learning and/or manual parameter tuning steps to boost their performance, and thus their generalization capabilities are questionable. Here we propose a novel methodology to compare IQA models. We first build a database that contains 4,744 source natural images, together with 94,880 distorted images created from them. We then propose a new mechanism, namely group MAximum Differentiation (gMAD) competition, which automatically selects subsets of image pairs from the database that provide the strongest test to let the IQA models compete with each other. Subjective testing on the selected subsets reveals the relative performance of the IQA models and provides useful insights on potential ways to improve them. We report the gMAD competition results between 16 well-known IQA models, but the framework is extendable, allowing future IQA models to be added into the competition.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Self-citationMaking a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766 · cited 4×
“The four distortion types are common in existing IQA databases [27, 23] and many IQA models are claimed to be able to properly handle these distortions [19, 20, 25, 17, 40, 18, 38, 37, 26, 42, 36, 7, 35].”
BackgroundMethodResultInfluentialMultiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253 · cited 2×
“Third, the best performance overall is obtained by MS-SSIM [34], which is a multi-scale version of SSIM [32] and significantly improves upon it.”
MethodSelf-citationNo-Reference Image Quality Assessment in the Spatial Domain
Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2214050 · Cited by 6,064 · cited 3×
“Fifth, a model that is worth noting is LPSI [36], which essentially reduces the feature space to one dimension and without using MOS for training, but it outperforms sophisticated machine learning-based approaches such as BRISQUE [17] and DIIVINE [20] which use many features for training.”
BackgroundMethodMatrix Analysis and Applied Linear Algebra
Meyer, C. (2000) · doi:10.1137/1.9780898719512 · Cited by 3,565
Mean squared error: Love it or leave it? A new look at Signal Fidelity Measures
Wang, Z., & Bovik, A. C. (2009) · IEEE Signal Processing Magazine · doi:10.1109/msp.2008.930649 · Cited by 3,175
Self-citationMost apparent distortion: full-reference image quality assessment and the role of strategy
Larson, E. C., & Chandler, D. M. (2010) · Journal of Electronic Imaging · doi:10.1117/1.3267105 · Cited by 1,953
Blind Image Quality Assessment: From Natural Scene Statistics to Perceptual Quality
Moorthy, A. K., & Bovik, A. C. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2147325 · Cited by 1,763
Image database TID2013: Peculiarities, results and perspectives
Ponomarenko, N., Jin, L., Ieremeiev, O., Lukin, V., Egiazarian, K., Astola, J., Vozel, B., Chehdi, K., Carli, M., Battisti, F., et al. (2015) · Signal Processing: Image Communication · doi:10.1016/j.image.2014.10.009 · Cited by 1,241 · cited 3×
“By contrast, the current database we created in this work contains 190 times more source images and 30 times more distorted images, respectively.”
BackgroundMethodInfluentialA Two-Step Framework for Constructing Blind Image Quality Indices
Moorthy, A. K., & Bovik, A. C. (2010) · IEEE Signal Processing Letters · doi:10.1109/lsp.2010.2043888 · Cited by 1,196
Cited by
Conditional Image Synthesis With Auxiliary Classifier GANs
Odena, A., Olah, C., & Shlens, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.09585 · Cited by 1,837
Blind Image Quality Assessment Using a Deep Bilinear Convolutional Neural Network
Zhang, W., Ma, K., Yan, J., Deng, D., & Wang, Z. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2886771 · Cited by 949
Self-citationBlindly Assess Image Quality in the Wild Guided by a Self-Adaptive Hyper Network
Su, S., Yan, Q., Zhu, Y., Zhang, C., Ge, X., Sun, J., & Zhang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00372 · Cited by 848
KonIQ-10k: An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment
Hosu, V., Lin, H., Sziranyi, T., & Saupe, D. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2967829 · Cited by 759
“Ma et al. [24] created a database with 4,744 pristine images and 94,880 distorted images to validate their proposed mechanism called group MAximum Differentiation (gMAD) competition.”
MethodEnd-to-End Blind Image Quality Assessment Using Deep Neural Networks
Ma, K., Liu, W., Zhang, K., Duanmu, Z., Wang, Z., & Zuo, W. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2774045 · Cited by 619
Self-citation
Metadata
- Authors
- Ma, K., Wu, Q., Wang, Z., Duanmu, Z., Yong, H., Li, H., & Zhang, L.
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 1664-1673
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2466606639
- Cited by
- 93
- References
- 13
- FWCI
- 4.23
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Group MAD Competition? A New Methodology to Compare Objective Image Quality Models. https://app.cochat.ai/papers/10.1109/cvpr.2016.184
- The paper
- Ma, K., Wu, Q., Wang, Z., Duanmu, Z., Yong, H., Li, H., & Zhang, L. (2016). Group MAD Competition? A New Methodology to Compare Objective Image Quality Models. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr.2016.184
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.