Cited by 963Generative Adversarial Networks and Image SynthesisDigital Media Forensic DetectionAdvanced Neural Network Applications

Pros and cons of GAN evaluation measures

Open access
Borji, Ali·2019·Computer Vision and Image Understanding

Abstract

Generative models , in particular generative adversarial networks (GANs), have gained significant attention in recent years. A number of GAN variants have been proposed and have been utilized in many applications. Despite large strides in terms of theoretical progress, evaluating and comparing GANs remains a daunting task. While several measures have been introduced, as of yet, there is no consensus as to which measure best captures strengths and limitations of models and should be used for fair model comparison. As in other areas of computer vision and machine learning , it is critical to settle on one or few good measures to steer the progress in this field. In this paper, I review and critically discuss more than 24 quantitative and 5 qualitative measures for evaluating generative models with a particular emphasis on GAN-derived models. I also provide a set of 7 desiderata followed by an evaluation of whether a given measure or a family of measures is compatible with them.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.cviu.2018.10.009

Cites

110 of 110 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The diversity and quality of GAN-generated images are evaluated manually by a physician or quantitatively using similarity measures, generally FID, structural similarity index, and peak signal-to-noise ratio.

    “The diversity and quality of the new images produced by GANs are evaluated manually by a physician or quantitatively by using similarity evaluation measurements, which are generally FID, structural similarity index, and peak signal-to-noise ratio (Borji 2019; Xu et al. 2018).”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. L. A. Gatys, A. S. Ecker, M. Bethge, Image style transfer using convo- lutional neural networks, in: Computer Vision and Pattern Recognition (CVPR), 2016 IEEE Conference on, IEEE, 2016, pp. 2414-2423.
  2. J. Johnson, A. Alahi, L. Fei-Fei, Perceptual losses for real-time style transfer and super-resolution, in: European Conference on Computer Vision, Springer, 2016, pp. 694-711.
  3. P. Isola, J.-Y. Zhu, T. Zhou, A. A. Efros, Image-to-image translation with conditional adversarial networks, arXiv preprint.
  4. C. Vondrick, H. Pirsiavash, A. Torralba, Generating videos with scene dynamics, in: Advances In Neural Information Processing Systems, 2016, pp. 613-621.
  5. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Advances in neural information processing systems, 2014, pp. 2672-2680.
  6. D. P. Kingma, S. Mohamed, D. J. Rezende, M. Welling, Semi-supervised learning with deep generative models, in: Advances in Neural Information Processing Systems, 2014, pp. 3581-3589.
  7. K. Kurach, M. Lucic, X. Zhai, M. Michalski, S. Gelly, The gan landscape: Losses, architectures, regularization, and normalization, arXiv preprint arXiv:1807.04720. · arXiv
  8. G. Huang, Y. Yuan, Q. Xu, C. Guo, Y. Sun, F. Wu, K. Weinberger, An empirical study on evaluation metrics of generative adversarial networks, International Conference on Learning RepresentationsRejected. URL https://openreview.net/forum?id=Sy1f0e-R-
  9. I. O. Tolstikhin, S. Gelly, O. Bousquet, C.-J. Simon-Gabriel, B. Schölkopf, Adagan: Boosting generative models, in: Advances in Neural Information Processing Systems, 2017, pp. 5430-5439.
  10. Z. Zhou, H. Cai, S. Rong, Y. Song, K. Ren, W. Zhang, J. Wang, Y. Yu, Activation maximization generative adversarial nets, International Confer- ence on Learning Representations. URL https://openreview.net/forum?id=HyyP33gAZ
  11. Z. Yi, H. Zhang, P. Tan, M. Gong, Dualgan: Unsupervised dual learning for image-to-image translation, arXiv preprint.
  12. X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, P. Abbeel, Infogan: Interpretable representation learning by information maximizing generative adversarial nets, in: Advances in Neural Information Processing Systems, 2016, pp. 2172-2180.
  13. M. F. Mathieu, J. J. Zhao, J. Zhao, A. Ramesh, P. Sprechmann, Y. LeCun, Disentangling factors of variation in deep representation using adversarial training, in: Advances in Neural Information Processing Systems, 2016, pp. 5040-5048.
  14. Z. Zhou, W. Zhang, J. Wang, Inception score, label smoothing, gradient vanishing and-log (d (x)) alternative, arXiv preprint arXiv:1708.01729. · arXiv
  15. C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, B. Póczos, Mmd gan: Towards deeper understanding of moment matching network, in: Advances in Neural Information Processing Systems, 2017, pp. 2200-2210.
  16. R. Zhang, P. Isola, A. A. Efros, Colorful image colorization, in: European Conference on Computer Vision, Springer, 2016, pp. 649-666.
  17. T. Lesort, F. Bordes, J.-F. Goudou, D. Filliat, Evaluation of generative networks through their data augmentation capacity (2018). URL https://openreview.net/forum?id=HJ1HFlZAb
  18. M. E. Glickman, A comprehensive guide to chess ratings, American Chess Journal 3 (1995) 59-102.
  19. S. Ghebreab, S. Scholte, V. Lamme, A. Smeulders, A biologically plausi- ble model for rapid natural scene identification, in: Advances in Neural Information Processing Systems, 2009, pp. 629-637.
  20. J. Snell, K. Ridgeway, R. Liao, B. D. Roads, M. C. Mozer, R. S. Zemel, Learning to generate images with perceptual similarity metrics, arXiv preprint arXiv:1511.06409. · arXiv
  21. L. v. d. Maaten, G. Hinton, Visualizing data using t-sne, Journal of machine learning research 9 (Nov) (2008) 2579-2605.
  22. D. J. Im, A. H. Ma, G. W. Taylor, K. Branson, Quantitatively evaluating gans with divergences proposed for training, International Conference on Learning RepresentationsAccepted as poster. URL https://openreview.net/forum?id=SJQHjzZ0-

Cited by

Sorted by citations

Authors
Borji, A.
Corresponding author
Borji, Ali
Venue
Computer Vision and Image Understanding
Publisher
Elsevier BV
Published
2019-02
Volume
179
Pages
41-65
ISSN
1077-3142 (ISSN-L)
OpenAlex
W2787289806
Cited by
963
References
110
Citations by year
2026: 46 · 2025: 113 · 2024: 121 · 2023: 161 · 2022: 138 · 2021: 182 · 2020: 123 · 2019: 58 · 2018: 21
FWCI
48.01
Type
journal article
Language
en
Open access
Locations
  • Computer Vision and Image Understanding (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
generative adversarial networks, GAN evaluation metrics, generative model evaluation, qualitative evaluation, model comparison, generative models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review