The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Exploring CLIP for Assessing the Look and Feel of Images
Open accessAbstract
Measuring the perception of visual content is a long-standing problem in computer vision. Many mathematical models have been developed to evaluate the look or quality of an image. Despite the effectiveness of such tools in quantifying degradations such as noise and blurriness levels, such quantification is loosely coupled with human language. When it comes to more abstract perception about the feel of visual content, existing methods can only rely on supervised models that are explicitly trained with labeled data collected via laborious user study. In this paper, we go beyond the conventional paradigms by exploring the rich visual language prior encapsulated in Contrastive Language-Image Pre-training (CLIP) models for assessing both the quality perception (look) and abstract perception (feel) of images without explicit task-specific training. In particular, we discuss effective prompt designs and show an effective prompt pairing strategy to harness the prior. We also provide extensive experiments on controlled datasets and Image Quality Assessment (IQA) benchmarks. Our results show that CLIP captures meaningful priors that generalize well to different perceptual assessments.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Making a “Completely Blind” Image Quality Analyzer
Mittal, A., Soundararajan, R., & Bovik, A. C. (2013) · IEEE Signal Processing Letters · doi:10.1109/lsp.2012.2227726 · Cited by 6,766
No-Reference Image Quality Assessment in the Spatial Domain
Mittal, A., Moorthy, A. K., & Bovik, A. C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2214050 · Cited by 6,064
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304
Self-citationNatural language processing: state of the art, current trends and challenges
Khurana, D., Koli, A., Khatter, K., & Singh, S. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13428-4 · Cited by 1,823
Blind Image Quality Assessment: A Natural Scene Statistics Approach in the DCT Domain
Saad, M. A., Bovik, A. C., & Charrier, C. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2191563 · Cited by 1,751
A Feature-Enriched Completely Blind Image Quality Evaluator
Zhang, L., Zhang, L., & Bovik, A. C. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2426416 · Cited by 1,311
Convolutional Neural Networks for No-Reference Image Quality Assessment
Kang, L., Ye, P., Li, Y., & Doermann, D. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.224 · Cited by 1,270
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.595 · Cited by 1,232
A Two-Step Framework for Constructing Blind Image Quality Indices
Moorthy, A. K., & Bovik, A. C. (2010) · IEEE Signal Processing Letters · doi:10.1109/lsp.2010.2043888 · Cited by 1,196
StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery
Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., & Lischinski, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00209 · Cited by 1,023
MUSIQ: Multi-scale Image Quality Transformer
Ke, J., Wang, Q., Wang, Y., Milanfar, P., & Yang, F. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00510 · Cited by 966
The statistics of natural images
Ruderman, D. (1994) · Network: Computation in Neural Systems · doi:10.1088/0954-898x/5/4/006 · Cited by 941
AVA: A large-scale database for aesthetic visual analysis
Murray, N., Marchesotti, L., & Perronnin, F. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247954 · Cited by 936
Blindly Assess Image Quality in the Wild Guided by a Self-Adaptive Hyper Network
Su, S., Yan, Q., Zhu, Y., Zhang, C., Ge, X., Sun, J., & Zhang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00372 · Cited by 848
KonIQ-10k: An Ecologically Valid Database for Deep Learning of Blind Image Quality Assessment
Hosu, V., Lin, H., Sziranyi, T., & Saupe, D. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2967829 · Cited by 759
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting
Rao, Y., Zhao, W., Chen, G., Tang, Y., Zhu, Z., Huang, G., Zhou, J., & Lu, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01755 · Cited by 631
End-to-End Blind Image Quality Assessment Using Deep Neural Networks
Ma, K., Liu, W., Zhang, K., Duanmu, Z., Wang, Z., & Zuo, W. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2774045 · Cited by 619
Learning a no-reference quality metric for single-image super-resolution
Ma, C., Yang, C.-Y., Yang, X., & Yang, M.-H. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2016.12.009 · Cited by 605
RegionCLIP: Region-based Language-Image Pretraining
Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01629 · Cited by 595
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language ModelingPreprint
Zhang, R., Fang, R., Zhang, W., Gao, P., Li, K., Dai, J., Qiao, Y., & Li, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03930 · Cited by 586
Learning photographic global tonal adjustment with a database of input/output image pairs
Bychkovsky, V., Paris, S., Chan, E., & Durand, F. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995413 · Cited by 536
Learning photographic global tonal adjustment with a database of input / output image pairs
Bychkovsky, V., Paris, S., Chan, E., & Durand, F. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995332 · Cited by 529
KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment
Hosu, V, Lin, H., Szirányi, T., & Saupe, D. (2020) · SZTAKI Publication Repository (Hungarian Academy of Sciences) · Cited by 512
Photo Aesthetics Ranking Network with Attributes and Content Adaptation
Kong, S., Shen, X., Lin, Z., Mech, R., & Fowlkes, C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_40 · Cited by 468
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis
Jain, A., Tancik, M., & Abbeel, P. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00583 · Cited by 463
RankSRGAN: Generative Adversarial Networks With Ranker for Image Super-Resolution
Zhang, W., Liu, Y., Dong, C., & Qiao, Y. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00319 · Cited by 405
Perceptual Quality Assessment of Smartphone Photography
Fang, Y., Zhu, H., Zeng, Y., Ma, K., & Wang, Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00373 · Cited by 375
Real-World Blur Dataset for Learning and Benchmarking Deblurring Algorithms
Rim, J., Lee, H., Won, J., & Cho, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58595-2_12 · Cited by 373
Real-world Noisy Image Denoising: A New BenchmarkPreprint
Xu, J., Li, H., Liang, Z., Zhang, D. D., & Zhang, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.02603 · Cited by 289
Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
Gu, X., Lin, T.-Y., Kuo, W., & Cui, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13921 · Cited by 273
ArtEmis: Affective Language for Visual Art
Achlioptas, P., Ovsjanikov, M., Haydarov, K., Elhoseiny, M., & Guibas, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01140 · Cited by 161
Attention-based Multi-Patch Aggregation for Image Aesthetic Assessment
Sheng, K., Dong, W., Ma, C., Mei, X., Huang, F., & Hu, B.-G. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240554 · Cited by 123
Positional Encoding as Spatial Inductive Bias in GANs
Xu, R., Wang, X., Chen, K., Zhou, B., & Loy, C. C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01336 · Cited by 108
Self-citationBuilding Emotional Machines: Recognizing Image Emotions Through Deep Neural Networks
Kim, H., Kim, Y.-S., Kim, S. J., & Lee, I. (2018) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2018.2827782 · Cited by 104
ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cues
Shi, H., Hayat, M., Wu, Y., & Cai, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00939 · Cited by 90
Investigating Loss Functions for Extreme Super-Resolution
Jo, Y., Yang, S., & Kim, S. J. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00220 · Cited by 77
Xu, Z., Lin, T., Tang, H., Li, F., He, D., Sebe, N., Timofte, R., Gool, L. V., & Ding, E. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01769 · Cited by 61
Learning the Non-differentiable Optimization for Blind Super-Resolution
Zheng, H., Li, J., Wang, X., & Gao, X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00213 · Cited by 41
An Image is Worth More Than a Thousand Words: Towards Disentanglement in the WildPreprint
Gabbay, A., Cohen, N., & Hoshen, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15610 · Cited by 24
DenseCLIP: Extract Free Dense Labels from CLIPPreprint
Zhou, C., Loy, C. C., & Dai, B. (2021) · arXiv (Cornell University) · Cited by 21
Self-citationAPSE: Attention-Aware Polarity-Sensitive Embedding for Emotion-Based Image Retrieval
Yao, X., Zhao, S., Lai, Y., She, D., Liang, J., & Yang, J. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3042664 · Cited by 17
Xu, Z., Lin, T., Tang, H., Li, F., He, D., Sebe, N., Timofte, R., Gool, L. V., & Ding, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.13333 · Cited by 3
Cited by
Exploiting Diffusion Prior for Real-World Image Super-Resolution
Wang, J., Yue, Z., Zhou, S., Chan, K. C. K., & Loy, C. C. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02168-7 · Cited by 439
Self-citationBlind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective
Zhang, W., Zhai, G., Wei, Y., Yang, X., & Ma, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01352 · Cited by 325
DiffBIR: Toward Blind Image Restoration with Generative Diffusion Prior
Lin, X., He, J., Chen, Z., Lyu, Z., Dai, B., Yu, F., Qiao, Y., Ouyang, W., & Dong, C. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-73202-7_25 · Cited by 266
SinSR: Diffusion-Based Image Super-Resolution in a Single Step
Wang, Y., Yang, W., Chen, X., Wang, Y., Guo, L., Chau, L.-P., Liu, Z., Qiao, Y., Kot, A. C., & Wen, B. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02437 · Cited by 196
PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
Feng, Y., Wang, X., Wong, K. K., Wang, S., Lu, Y., Zhu, M., Wang, B., & Chen, W. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327168 · Cited by 124
Metadata
- Authors
- Wang, J., Chan, K. C. K., & Loy, C. C.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2023-06-26
- Volume
- 37
- Issue
- 2
- Pages
- 2555-2563
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4382462760
- Cited by
- 733
- References
- 43
- FWCI
- 83.35
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.