Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 10

The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Zhang, Richard, Isola, Phillip, Efros, Alexei A., Shechtman, Eli, Wang, Oliver·2018·arXiv
DOI 10.48550/arxiv.1801.03924arXiv 1801.03924
Open this paper in CoChat — free account

Abstract

While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called "perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1801.03924

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 2 references

Research works printed but not linked: 48

  1. P. Agrawal, J. Carreira, and J. Malik. Learning to see by moving. In ICCV, pages 37-45, 2015. 7
  2. E. Agustsson and R. Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In CVPR Work- shops, July 2017. 4
  3. S. Ali Amirshahi, M. Pedersen, and S. X. Yu. Image qual- ity assessment by comparing cnn features between images. Electronic Imaging, 2017(12):42-51, 2017. 3
  4. J. R. Anderson. The adaptive character of thought. Psychol- ogy Press, 1990. 8
  5. A. Berardino, V. Laparra, J. Ballé, and E. Simoncelli. Eigen- distortions of hierarchical representations. In NIPS, 2017. 3
  6. V. Bychkovsky, S. Paris, E. Chan, and F. Durand. Learn- ing photographic global tonal adjustment with a database of input / output image pairs. In CVPR, 2011. 3, 5
  7. Q. Chen and V. Koltun. Photographic image synthesis with cascaded refinement networks. ICCV, 2017. 2, 5
  8. D.-T. Dang-Nguyen, C. Pasquini, V. Conotter, and G. Boato. Raise: a raw images dataset for digital image forensics. In Proceedings of the 6th ACM Multimedia Systems Confer- ence, pages 219-224. ACM, 2015. 3, 5
  9. M. Delbracio and G. Sapiro. Hand-held video deblurring via efficient fourier aggregation. IEEE Transactions on Compu- tational Imaging, 1(4):270-283, 2015. 4
  10. C. Doersch, A. Gupta, and A. A. Efros. Unsupervised vi- sual representation learning by context prediction. In ICCV, 2015. 7, 8
  11. J. Donahue, P. Krähenbühl, and T. Darrell. Adversarial fea- ture learning. ICLR, 2017. 1, 2, 5, 7, 8, 12, 14
  12. A. Dosovitskiy and T. Brox. Generating images with per- ceptual similarity metrics based on deep networks. In HIPS, pages 658-666, 2016. 2, 5
  13. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal- network.org/challenges/VOC/voc2007/workshop/index.html. 7
  14. L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In CVPR, pages 2414- 2423, 2016. 2, 5
  15. F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < 0.5 mb model size. CVPR, 2017. 1, 2, 5, 7, 12
  16. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. CVPR, 2017. 5
  17. P. Isola, D. Zoran, D. Krishnan, and E. H. Adelson. Learning visual groups from co-occurrences in space and time. ICCV Workshop, 2016. 4
  18. J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. ECCV, 2016. 2
  19. J. Kim, J. Kwon Lee, and K. Mu Lee. Accurate image super- resolution using very deep convolutional networks. In CVPR, pages 1646-1654, 2016. 4
  20. J. Kim and S. Lee. Deep learning of human visual sensitivity in image quality assessment framework. In CVPR, 2017. 3
  21. P. Krähenbühl, C. Doersch, J. Donahue, and T. Darrell. Data-dependent initializations of convolutional neural net- works. International Conference on Learning Representa- tions, 2016. 1, 2, 7, 12
  22. A. Krizhevsky. One weird trick for parallelizing convo- lutional neural networks. arXiv preprint arXiv:1404.5997, 2014. 1, 5, 7, 12, 14 · arXiv
  23. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 2, 5, 6
  24. E. C. Larson and D. M. Chandler. Most apparent distortion: full-reference image quality assessment and the role of strat- egy. Journal of Electronic Imaging, 2010. 3
  25. G. Larsson, M. Maire, and G. Shakhnarovich. Learning rep- resentations for automatic colorization. ECCV, 2016. 4
  26. C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. Photo-realistic single image super-resolution using a genera- tive adversarial network. CVPR, 2016. 4
  27. B. Lim, S. Son, H. Kim, S. Nah, and K. M. Lee. Enhanced deep residual networks for single image super-resolution. In CVPR Workshops, 2017. 5
  28. C. Liu et al. Beyond pixels: exploring new representations and applications for motion analysis. PhD thesis, Mas- sachusetts Institute of Technology, 2009. 4
  29. R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich. Hdr- vdp-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. In ACM Transac- tions on Graphics (TOG), 2011. 1, 12
  30. A. B. Markman and D. Gentner. Nonintentional similarity processing. The new unconscious, pages 107-137, 2005. 2
  31. D. L. Medin, R. L. Goldstone, and D. Gentner. Respects for similarity. Psychological review, 100(2):254, 1993. 2, 5
  32. S. Meyer, O. Wang, H. Zimmer, M. Grosse, and A. Sorkine- Hornung. Phase-based frame interpolation for video. In CVPR, pages 1410-1418, 2015. 4
  33. N. Murray, L. Marchesotti, and F. Perronnin. Ava: A large- scale database for aesthetic visual analysis. In CVPR, 2012. 3
  34. S. Niklaus, L. Mai, and F. Liu. Video frame interpolation via adaptive separable convolution. In ICCV, 2017. 4
  35. M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV, 2016. 1, 2, 5, 7, 12
  36. A. Owens, P. Isola, J. McDermott, A. Torralba, E. H. Adel- son, and W. T. Freeman. Visually indicated sounds. CVPR, 2016. 7
  37. D. Pathak, R. Girshick, P. Dollár, T. Darrell, and B. Hariha- ran. Learning features by watching objects move. CVPR, 2017. 1, 5, 7, 12
  38. D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016. 7
  39. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 2015. 1, 4, 5
  40. M. S. Sajjadi, B. Schölkopf, and M. Hirsch. Enhancenet: Single image super-resolution through automated texture synthesis. ICCV, 2017. 4
  41. D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV, 2002. 4, 5
  42. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv, 2014. 1, 2, 5, 7, 12
  43. S. Su, M. Delbracio, J. Wang, G. Sapiro, W. Heidrich, and O. Wang. Deep video deblurring for hand-held cameras. In CVPR, 2017. 4
  44. A. Tversky. Features of similarity. Psychological review, 1977. 2
  45. X. Wang and A. Gupta. Unsupervised learning of visual rep- resentations using videos. In ICCV, 2015. 7
  46. Z. Wang, D. Liu, J. Yang, W. Han, and T. Huang. Deep networks for image super-resolution with sparse prior. In ICCV, 2015. 4
  47. R. Zhang, P. Isola, and A. A. Efros. Colorful image coloriza- tion. ECCV, 2016. 4, 5, 7
  48. R. Zhang, P. Isola, and A. A. Efros. Split-brain autoen- coders: Unsupervised learning by cross-channel prediction. In CVPR, 2017. 1, 2, 5, 7, 12

Cited by

Metadata

Authors
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O.
Venue
arXiv
Publisher
arXiv
Published
2018
DOI
10.48550/arxiv.1801.03924
arXiv
1801.03924
References
2
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review