CoChat

Waterloo Exploration Database: New Challenges for Image Quality Assessment Models

Ma, Kede, Duanmu, Zhengfang, Wu, Qingbo, Wang, Zhou, Yong, Hongwei, Li, Hongliang, Zhang, Lei · 2017 · IEEE Transactions on Image Processing

The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.

Abstract

The great content diversity of real-world digital images poses a grand challenge to image quality assessment (IQA) models, which are traditionally designed and validated on a handful of commonly used IQA databases with very limited content variation. To test the generalization capability and to facilitate the wide usage of IQA techniques in real-world applications, we establish a large-scale database named the Waterloo Exploration Database, which in its current state contains 4744 pristine natural images and 94 880 distorted images created from them. Instead of collecting the mean opinion score for each image via subjective testing, which is extremely difficult if not impossible, we present three alternative test criteria to evaluate the performance of IQA models, namely, the pristine/distorted image discriminability test, the listwise ranking consistency test, and the pairwise preference consistency test (P-test). We compare 20 well-known IQA models using the proposed criteria, which not only provide a stronger test in a more challenging testing environment for existing models, but also demonstrate the additional benefits of using the proposed database. For example, in the P-test, even for the best performing no-reference IQA model, more than 6 million failure cases against the model are "discovered" automatically out of over 1 billion test pairs. Furthermore, we discuss how the new database may be exploited using innovative approaches in the future, to reveal the weaknesses of existing IQA models, to provide insights on how to improve the models, and to shed light on how the next-generation IQA models may be developed. The database and codes are made publicly available at: https://ece.uwaterloo.ca/~k29ma/exploration/.

Overview

Integrity signals

Registry

Crossref · 10.1109/tip.2016.2631888

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Citations

Cites

Show all 43 references

Research works printed but not linked: 5

  1. ponomarenko, 2008, Tampere Image Database TID2008
  2. winkler, 2016, Image and video quality resources
  3. sheikh, 2016, Image and Video Quality Assessment Research at LIVE
  4. 2000, Final report from the video quality experts group on the validation of objective models of video quality assessment
  5. horita, 2010, Toyama-MICT Image Quality Evaluation Database

Cited by

See all 789

Metadata

Authors
Ma, K., Duanmu, Z., Wu, Q., Wang, Z., Yong, H., Li, H., & Zhang, L.
Affiliations
  • University of Waterloo (CA)
  • University of Electronic Science and Technology of China (CN)
  • Hong Kong Polytechnic University (HK)
Venue
IEEE Transactions on Image Processing
Published
2017-02
Volume
26
Issue
2
Pages
1004-1016
ISSN
  • 1057-7149 (ISSN-L)
  • 1941-0042
OpenAlex
W2556068545
Cited by
789
References
43
Citations by year
2026: 90 · 2025: 123 · 2024: 106 · 2023: 110 · 2022: 70 · 2021: 92 · 2020: 78 · 2019: 69 · 2018: 35 · 2017: 16
FWCI
26.97
Type
journal article
Language
en
Open access
closed
Locations
  • IEEE Transactions on Image Processing (publishedVersion)
  • PubMed (publishedVersion)
  • PolyU Institutional Research Archive (Hong Kong Polytechnic University) (submittedVersion)
Funders
  • National Natural Science Foundation of China
  • China Scholarship Council
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Keywords
image quality assessment, no-reference image quality assessment, mean opinion score

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.