CoChat
Sign Up
PreprintCited by 732Generative Adversarial Networks and Image SynthesisAdvanced Neural Network ApplicationsAdversarial Robustness in Machine Learning

The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.

A Note on the Inception Score

Open access
Barratt, Shane, Sharma, Rishi·2018·arXiv
DOI 10.48550/arxiv.1801.01973arXiv 1801.01973
Open this paper in CoChat — free accountOpen PDF

Abstract

Deep generative models are powerful tools that have produced impressive results in recent years. These advances have been for the most part empirically driven, making it essential that we use high quality evaluation metrics. In this paper, we provide new insights into the Inception Score, a recently proposed and widely used evaluation metric for generative models, and demonstrate that it fails to provide useful guidance when comparing models. We discuss both suboptimalities of the metric itself and issues with its application. Finally, we call for researchers to be more systematic and careful when evaluating and comparing generative models, as the advancement of the field depends upon it.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1801.01973

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 3 references

Research works printed but not linked: 20

  1. Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017. · arXiv
  2. Che, T., Li, Y., Jacob, A. P., Bengio, Y., and Li, W. Mode reg- ularized generative adversarial networks. arXiv preprint arXiv:1612.02136, 2016. · arXiv
  3. Esteban, C., Hyland, S. L., and Rätsch, G. Real-valued (Medical) Time Series Generation with Recurrent Condi- tional GANs. ArXiv e-prints, June 2017.
  4. Farimani, A. B., Gomes, J., and Pande, V. S. Deep learning the physics of transport phenomena. CoRR, abs/1709.02432, 2017. URL http://arxiv.org/ abs/1709.02432. · arXiv
  5. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680, 2014a.
  6. Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014b. · arXiv
  7. Guo, J., Lu, S., Cai, H., Zhang, W., Yu, Y., and Wang, J. Long text generation via adversarial training with leaked information. CoRR, abs/1709.08624, 2017. URL http://arxiv.org/abs/1709.08624. · arXiv
  8. Karras, T., Aila, T., Laine, S., and Lehtinen, J. Progres- sive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017. · arXiv
  9. Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A. P., Tejani, A., Totz, J., Wang, Z., and Shi, W. Photo-realistic single image super-resolution using a generative adver- sarial network. CoRR, abs/1609.04802, 2016. URL http://arxiv.org/abs/1609.04802. · arXiv
  10. Lucic, M., Kurach, K., Michalski, M., Gelly, S., and Bous- quet, O. Are gans created equal? a large-scale study. arXiv preprint arXiv:1711.10337, 2017. · arXiv
  11. Mogren, O. C-RNN-GAN: continuous recurrent neural net- works with adversarial training. CoRR, abs/1611.09904, 2016. URL http://arxiv.org/abs/1611. 09904. · arXiv
  12. Qiantong, X., Gao, H., Yang, Y., Chuan, G., Yu, S., Felix, W., and Weinberger, K. An empirical study on evaluation metrics of generative adversarial networks. arXiv preprint arXiv:1806.07755, 2018. · arXiv
  13. Rosca, M., Lakshminarayanan, B., Warde-Farley, D., and Mohamed, S. Variational approaches for auto- encoding generative adversarial networks. arXiv preprint arXiv:1706.04987, 2017. · arXiv
  14. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition chal- lenge. International Journal of Computer Vision, 115(3): 211-252, 2015.
  15. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Rad- ford, A., and Chen, X. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 2234-2242, 2016.
  16. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R. Intriguing properties of neural networks. CoRR, abs/1312.6199, 2013. URL http://arxiv.org/abs/1312.6199. · arXiv
  17. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 2818-2826, 2016.
  18. Theis, L., Oord, A. v. d., and Bethge, M. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01844, 2015. · arXiv
  19. van den Oord, A. and Dambre, J. Locally-connected trans- formations for deep gmms. In International Conference on Machine Learning (ICML): Deep learning Workshop, pp. 1-8, 2015.
  20. Zhu, J., Park, T., Isola, P., and Efros, A. A. Unpaired image- to-image translation using cycle-consistent adversarial networks. CoRR, abs/1703.10593, 2017. URL http: //arxiv.org/abs/1703.10593. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Barratt, S., & Sharma, R.
Affiliations
Stanford University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1801.01973
arXiv
1801.01973
OpenAlex
W2782980316
Cited by
732
References
3
Citations by year
2026: 47 · 2025: 123 · 2024: 84 · 2023: 94 · 2022: 69 · 2021: 111 · 2020: 94 · 2019: 69 · 2018: 41
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (DGE-165651 8, 1656518)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
Inception Score, generative model evaluation, evaluation metrics, model comparison, deep generative models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review