The DOI matches its Crossref record; 44 of the 46 references it cites were found in a registry or an index.
Learning to generate images with perceptual similarity metrics
Open accessAbstract
Deep networks are increasingly being applied to problems involving image synthesis, e.g., generating images from textual descriptions and reconstructing an input image from a compact representation. Supervised training of image-synthesis networks typically uses a pixel-wise loss (PL) to indicate the mismatch between a generated image and its corresponding target image. We propose instead to use a loss function that is better calibrated to human perceptual judgments of image quality: the multiscale structural-similarity score (MS-SSIM) [1]. Because MS-SSIM is differentiable, it is easily incorporated into gradient-descent learning. We compare the consequences of using MS-SSIM versus PL loss on training autoencoders. Human observers reliably prefer images synthesized by MS-SSIM-optimized models over those synthesized by PL-optimized models, for two distinct PL measures (L1and L2distances). We also explore the effect of training objective on image encoding and analyze conditions under which perceptually-optimized representations yield better performance on image classification. Finally, we demonstrate the superiority of perceptually-optimized networks for super-resolution imaging. We argue that significant advances can be made in modeling images through the use of training objectives that are well aligned to characteristics of human perception.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Conversely, modern deep-learning metrics like LPIPS offer better perceptual alignment but remain computationally prohibitive for real-time, resource-constrained environments.”
From A Survey and Tutorial on Image Quality Assessment with a Contrast-Weighted Structural Similarity Framework (2026) · cites this paper as [30]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Deep learning-based similarity [29-34]: Semantic similarity learning uses convolutional neural networks (CNNs) to embed images in a structured feature space. Metrics such as LPIPS (learned perceptual image patch similarity) correlate strongly with human judgment but incur high computational costs, offer low interpretability (a 'black-box' nature), and are vulnerable to adversarial perturbations.”
From A Survey and Tutorial on Image Quality Assessment with a Contrast-Weighted Structural Similarity Framework (2026) · cites this paper as [30]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Image Super-Resolution Using Deep Convolutional Networks
Dong, C., Loy, C. C., He, K., & Tang, X. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2439281 · Cited by 10,105
Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2002) · Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001 · doi:10.1109/iccv.2001.937655 · Cited by 8,216
Multiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253
Image information and visual quality
Sheikh, H. R., & Bovik, A. C. (2006) · IEEE Transactions on Image Processing · doi:10.1109/tip.2005.859378 · Cited by 4,104
On Single Image Scale-Up Using Sparse-Representations
Zeyde, R., Elad, M., & Protter, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-27413-8_47 · Cited by 3,371
Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding
Bevilacqua, M., Roumy, A., Guillemot, C. M., & Morel, M.-L. A. (2012) · British Machine Vision Conference (BMVC) · doi:10.5244/c.26.135 · Cited by 3,162
An analysis of single-layer networks in unsupervised feature learning
Coates, A., Ng, A. Y., & Lee, H. (2011) · Cited by 2,760
Generative Adversarial Networks
Drori, I. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108891530.013 · Cited by 2,557
Acquiring linear subspaces for face recognition under variable lighting
Lee, K.-C., Ho, J., & Kriegman, D. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.92 · Cited by 2,435
Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction
Masci, J., Meier, U., Ciresan, D. C., & Schmidhuber, J. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-21735-7_7 · Cited by 2,137
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
Torralba, A., Fergus, R., & Freeman, W. T. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.128 · Cited by 1,921
Deep Generative Image Models using a Laplacian Pyramid of Adversarial NetworksPreprint
Denton, E., Chintala, S., Szlam, A. D., & Fergus, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.05751 · Cited by 1,789
Information processing in dynamical systems: foundations of harmony theory
Smolensky, P. (1986) · The MIT Press eBooks · Cited by 1,673
DRAW: A Recurrent Neural Network For Image GenerationPreprint
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.04623 · Cited by 1,498
Forward models: supervised learning with a distal teacher
Jordan, M. I., & Rumelhart, D. E. (1992) · Cognitive Science · doi:10.1207/s15516709cog1603_1 · Cited by 1,451
Dayan, P., Hinton, G. E., Neal, R. M., & Zemel, R. S. (1995) · Neural Computation · doi:10.1162/neco.1995.7.5.889 · Cited by 1,207
Self-citationVSNR: A Wavelet-Based Visual Signal-to-Noise Ratio for Natural Images
Chandler, D. M., & Hemami, S. S. (2007) · IEEE Transactions on Image Processing · doi:10.1109/tip.2007.901820 · Cited by 1,147
Learning and relearning in Boltzmann machines
Hinton, G. E., & Sejnowski, T. J. (2001) · Graphical Models · doi:10.7551/mitpress/3349.003.0005 · Cited by 1,037
A note on the evaluation of generative models
Theis, L., Oord, A. V. D., & Bethge, M. (2015) · arXiv · doi:10.48550/arxiv.1511.01844 · Cited by 1,026
Multiscale structural similarity for image quality assessment
Wang, Z. (2025) · doi:10.57702/c0fj14hf · Cited by 875
The visible differences predictor: an algorithm for the assessment of image fidelity
Daly, S. J. (1993) · The MIT Press eBooks · Cited by 828
Visible differences predictor: an algorithm for the assessment of image fidelity
Daly, S. J. (1992) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.135952 · Cited by 718
Generative Moment Matching NetworksPreprint
Li, Y., Swersky, K., & Zemel, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.02761 · Cited by 657
Self-citationUsing very deep autoencoders for content-based image retrieval.
Krizhevsky, A., & Hinton, G. E. (2011) · Cited by 372
Perceptual quality measure using a spatiotemporal model of the human visual system
Lambrecht, C. J. V. D. B., & Verscheure, O. (1996) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.235440 · Cited by 340
Loss Functions for Neural Networks for Image ProcessingPreprint
Zhao, H., Gallo, O., Frosio, I., & Kautz, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.08861 · Cited by 236
Wang, Z., & Simoncelli, E. P. (2008) · Journal of Vision · doi:10.1167/8.12.8 · Cited by 138
Quaternion Structural Similarity: A New Quality Index for Color Images
Kolaman, A., & Yadid-Pecht, O. (2011) · IEEE Transactions on Image Processing · doi:10.1109/tip.2011.2181522 · Cited by 138
On Rate Distortion Optimization Using SSIM
Yeo, C., Tan, H. L., & Tan, Y. H. (2013) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2013.2240918 · Cited by 74
Structural Similarity Measure for Color Images
Hassan, A. M., & Bhagvati, C. (2012) · International Journal of Computer Applications · doi:10.5120/6169-8590 · Cited by 73
A Test of Relative Similarity For Model Selection in Generative ModelsPreprint
Bounliphone, W., Belilovsky, E., Blaschko, M. B., Antonoglou, I., & Gretton, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04581 · Cited by 60
A perceptual distortion metric for digital color images
WINKLER, S. S. (2002) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.1998.999029 · Cited by 50
Methodology for designing image similarity metrics based on human visual system models
Frese, T., Bouman, C. A., & Allebach, J. P. (1997) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.274545 · Cited by 43
Is L2 a Good Loss Function for Neural Networks for Image ProcessingPreprint
Zhao, H., Gallo, O., Frosio, I., & Kautz, J. (2015) · arXiv (Cornell University) · Cited by 36
Maximum a Posterior and Perceptually Motivated Reconstruction Algorithm: A Generic Framework
Jakhetiya, V., Lin, W., Jaiswal, S. P., Guntuku, S. C., & Au, O. C. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2609419 · Cited by 27
A Human Vision System Model For Objective Image Fidelity And Target Detectability Measurements
Lubin, J. S. (1998) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.36473 · Cited by 4
Research works printed but not linked: 1
- 10.1016/0364-0213(92)90036-T · DOI · Paper page
Cited by
Sorted by citations
Autoencoding beyond pixels using a learned similarity metric
Larsen, A., Sønderby, S. K., Larochelle, H., & Winther, O. (2016) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 1,317
Defense Against Adversarial Attacks Using High-Level Representation Guided Denoiser
Liao, F., Liang, M., Dong, Y., Pang, T., Hu, X., & Zhu, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00191 · Cited by 987
Pros and cons of GAN evaluation measures
Borji, A. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2018.10.009 · Cited by 963
High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis
Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., & Li, H. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.434 · Cited by 894
Improving Unsupervised Defect Segmentation by Applying Structural Similarity to Autoencoders
Bergmann, P., Löwe, S., Fauser, M., Sattlegger, D., & Steger, C. T. (2019) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0007364503720380 · Cited by 893
Metadata
- Authors
- Snell, J., Ridgeway, K., Liao, R., Roads, B. D., Mozer, M. C., & Zemel, R. S.
- Affiliations
- University of Toronto (CA)
- University of Colorado Boulder (US)
- Venue
- 2017 IEEE International Conference on Image Processing (ICIP)
- Publisher
- IEEE
- Published
- 2017-09
- Pages
- 4277-4281
- ISSN
- 1522-4880 (ISSN-L)
- OpenAlex
- W2174358748
- Semantic Scholar
- 90e4f12fa8fb126d0f416c8b61bfb5f73f8b7b74
- Cited by
- 186
- References
- 46
- Citations by year
- 2026: 15 · 2025: 23 · 2024: 21 · 2023: 21 · 2022: 17 · 2021: 22 · 2020: 28 · 2019: 16 · 2018: 11 · 2017: 7
- FWCI
- 4.96
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - International Conference on Image Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (0542013, 1041755, 1461535)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
- Keywords
- perceptual image quality, MS-SSIM, image generation, loss function design, autoencoder, image reconstruction, image classification, super-resolution, human perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
