CoChat
Sign Up
PreprintCited by 293Generative Adversarial Networks and Image SynthesisImage and Video Quality AssessmentAdvanced Image Processing Techniques

The DOI matches its DataCite record.

On Aliased Resizing and Surprising Subtleties in GAN Evaluation

Open access
Parmar, Gaurav, Zhang, Richard, Zhu, Jun-Yan·2021·arXiv
DOI 10.48550/arxiv.2104.11222arXiv 2104.11222
Open this paper in CoChat — free accountOpen PDF

Abstract

Metrics for evaluating generative models aim to measure the discrepancy between real and generated images. The often-used Frechet Inception Distance (FID) metric, for example, extracts "high-level" features using a deep network from the two sets. However, we find that the differences in "low-level" preprocessing, specifically image resizing and compression, can induce large variations and have unforeseen consequences. For instance, when resizing an image, e.g., with a bilinear or bicubic kernel, signal processing principles mandate adjusting prefilter width depending on the downsampling factor, to antialias to the appropriate bandwidth. However, commonly-used implementations use a fixed-width prefilter, resulting in aliasing artifacts. Such aliasing leads to corruptions in the feature extraction downstream. Next, lossy compression, such as JPEG, is commonly used to reduce the file size of an image. Although designed to minimally degrade the perceptual quality of an image, the operation also produces variations downstream. Furthermore, we show that if compression is used on real training images, FID can actually improve if the generated images are also subsequently compressed. This paper shows that choices in low-level image processing have been an underappreciated aspect of generative modeling. We identify and characterize variations in generative modeling development pipelines, provide recommendations based on signal processing principles, and release a reference implementation to facilitate future comparisons.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2104.11222

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 80

  1. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghe- mawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), pages 265-283, 2016.
  2. Sanjeev Arora and Yi Zhang. Do gans actually learn the distribution? an empirical study. In International Conference on Learning Representations (ICLR), 2018.
  3. Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transforma- tions? Journal of Machine Learning Research, 20:1-25, 2019.
  4. Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021.
  5. Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In ICLR, 2018.
  6. Gary Bradski and Adrian Kaehler. Opencv. Dr. Dobb's journal of software tools, 3, 2000.
  7. Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
  8. Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
  9. Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola. What makes fake images detectable? understanding properties that generalize. In European Conference on Computer Vision, pages 103-120. Springer, 2020.
  10. Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Min- jie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. In Advances in Neural Information Processing Systems, 2015.
  11. Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  12. Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them. In CVPR, 2020.
  13. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2009.
  14. Emily L Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adver- sarial networks. In Advances in Neural Information Process- ing Systems, 2015.
  15. Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Den- sity estimation using real nvp. In International Conference on Learning Representations (ICLR), 2017.
  16. Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, 2019.
  17. Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In Advances in Neural Information Processing Systems, 2016.
  18. Yilun Du and Igor Mordatch. Implicit generation and gen- eralization in energy-based models. In Advances in Neural Information Processing Systems, 2019.
  19. Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. In International Conference on Machine Learning, pages 1802-1811. PMLR, 2019.
  20. James D Foley, Foley Dan Van, Andries Van Dam, Steven K Feiner, John F Hughes, and J Hughes. Computer graphics: principles and practice, volume 12110. Addison-Wesley Professional, 1996.
  21. James D Foley, Andries Van Dam, Steven K Feiner, John F Hughes, and Richard L Phillips. Introduction to computer graphics, volume 55. Addison-Wesley Reading, 1994.
  22. David A Forsyth and Jean Ponce. Computer vision: a modern approach. Pearson,, 2012.
  23. Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Im- age style transfer using convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2016.
  24. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
  25. Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. ICLR, 2019.
  26. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bern- hard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
  27. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco An- dreetto, and Hartwig Adam. Mobilenets: Efficient convolu- tional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. · arXiv
  28. Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. Euro- pean Conference on Computer Vision (ECCV), 2018.
  29. Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision (ECCV), 2016.
  30. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Rep- resentations (ICLR), 2018.
  31. Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adver- sarial networks with limited data. NIPS, 33, 2020.
  32. Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In Proc. NeurIPS, 2021.
  33. Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2019.
  34. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  35. R. Keys. Cubic convolution interpolation for digital image processing. IEEE Transactions on Acoustics, Speech, and Signal Processing, 29(6):1153-1160, 1981.
  36. Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, 2018.
  37. Diederik P Kingma and Max Welling. Auto-encoding vari- ational bayes. International Conference on Learning Repre- sentations (ICLR), 2014.
  38. Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, 2019.
  39. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recog- nition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
  40. Erich L Lehmann and Joseph P Romano. Testing statistical hypotheses. Springer Science & Business Media, 2006.
  41. Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In IEEE International Confer- ence on Computer Vision (ICCV), 2019.
  42. Steven Liu, Tongzhou Wang, David Bau, Jun-Yan Zhu, and Antonio Torralba. Diverse image generation via self- conditioned gans. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  43. David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In ICLR, 2017.
  44. Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In Advances in Neural Information Processing Systems, 2018.
  45. Julien Mairal, Piotr Koniusz, Zaid Harchaoui, and Cordelia Schmid. Convolutional kernel networks. Advances in neural information processing systems, 27:2627-2635, 2014.
  46. Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of mcmc-based maximum likelihood learning of energy-based models. In AAAI Confer- ence on Artificial Intelligence (AAAI), 2020.
  47. Harry Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Institute of Electrical Engineers, 47(2):617-644, 1928.
  48. Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelcnn decoders. In Advances in Neural Information Processing Systems, 2016.
  49. Alan V. Oppenheim, Ronald W. Schafer, and John R. Buck. Discrete-Time Signal Processing. Pearson, 2nd edition, 1999.
  50. Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive nor- malization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  51. Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), 2021.
  52. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
  53. Shengju Qian, Hao Shao, Yi Zhu, Mu Li, and Jiaya Jia. Blend- ing anti-aliasing into vision transformer. In Thirty-Fifth Con- ference on Neural Information Processing Systems, 2021.
  54. Erwin Quiring, David Klein, Daniel Arp, Martin Johns, and Konrad Rieck. Adversarial preprocessing: Understanding and preventing image-scaling attacks in machine learning. In 29th {USENIX} Security Symposium ({USENIX} Security 20), pages 1363-1380, 2020.
  55. Alec Radford, Luke Metz, and Soumith Chintala. Unsuper- vised representation learning with deep convolutional gener- ative adversarial networks. In International Conference on Learning Representations (ICLR), 2016.
  56. Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In Advances in Neural Information Processing Systems, 2019.
  57. Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Gen- erating diverse high-fidelity images with vq-vae-2. In NIPS, 2019.
  58. Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, 2018.
  59. Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statistics, pages 448- 455, 2009.
  60. Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, 2016.
  61. Dominik Scherer, Andreas Müller, and Sven Behnke. Evalua- tion of pooling operations in convolutional architectures for object recognition. In International conference on artificial neural networks, pages 92-101. Springer, 2010.
  62. Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Josh Susskind, Wenda Wang, and Russ Webb. Learning from simulated and unsupervised images through adversarial train- ing. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  63. Laurent Sifre and Stéphane Mallat. Rigid-motion scattering for texture classification. arXiv preprint arXiv:1403.1687, 2014. · arXiv
  64. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception archi- tecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  65. Richard Szeliski. Computer vision: algorithms and applica- tions. Springer Science & Business Media, 2010.
  66. Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. In Ad- vances in Neural Information Processing Systems (NeurIPS), 2020.
  67. Lucas Theis, Aäron van den Oord, and Matthias Bethge. A note on the evaluation of generative models. In ICLR, 2016.
  68. Gregory K Wallace. The jpeg still picture compression stan- dard. IEEE transactions on consumer electronics, 38(1):xviii- xxxiv, 1992.
  69. Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros. Cnn-generated images are sur- prisingly easy to spot... for now. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  70. Qixue Xiao, Yufei Chen, Chao Shen, Yu Chen, and Kang Li. Seeing is not believing: Camouflage attacks on image scaling algorithms. In 28th {USENIX} Security Symposium ({USENIX} Security 19), pages 443-460, 2019.
  71. Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015. · arXiv
  72. Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In International Conference on Learning Repre- sentations (ICLR), 2020.
  73. Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In International Conference on Learning Repre- sentations (ICLR), 2020.
  74. Richard Zhang. Making convolutional networks shift- invariant again. In International Conference on Machine Learning (ICML), 2019.
  75. Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  76. Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan train- ing. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
  77. Sharon Zhou, Mitchell L Gordon, Ranjay Krishna, Austin Narcomey, Li Fei-Fei, and Michael S Bernstein. Hype: A benchmark for human eye perceptual evaluation of genera- tive models. In Advances in Neural Information Processing Systems, 2019.
  78. Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipulation on the natu- ral image manifold. In European Conference on Computer Vision (ECCV), 2016.
  79. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision (ICCV), 2017.
  80. Xueyan Zou, Fanyi Xiao, Zhiding Yu, and Yong Jae Lee. Delving deeper into anti-aliasing in convnets. In BMVC, 2020.

Cited by

Sorted by citations

Metadata

Authors
Parmar, G., Zhang, R., & Zhu, J.-Y.
Affiliations
  • Carnegie Mellon University (US)
  • Adobe Systems (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Pages
11400-11410
ISSN
1063-6919 (ISSN-L)
DOI
10.48550/arxiv.2104.11222
arXiv
2104.11222
OpenAlex
W4312694728
Cited by
293
References
0
Citations by year
2026: 42 · 2025: 74 · 2024: 106 · 2023: 53 · 2022: 18
FWCI
22.76
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image Processing Techniques
Keywords
generative adversarial networks, GAN evaluation metrics, Fréchet Inception Distance, image scaling, aliasing, downsampling, image compression, JPEG compression, image quality assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review