CoChat
Sign Up
PreprintCited by 8,365Generative Adversarial Networks and Image SynthesisStochastic Gradient Optimization TechniquesAdversarial Robustness in Machine Learning

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Open access
Heusel, Martin, Ramsauer, Hubert, Unterthiner, Thomas, Nessler, Bernhard, Hochreiter, Sepp·2017·arXiv
DOI 10.48550/arxiv.1706.08500arXiv 1706.08500
Open this paper in CoChat — free accountOpen PDF

Abstract

Generative Adversarial Networks (GANs) excel at creating realistic images with complex models for which maximum likelihood is infeasible. However, the convergence of GAN training has still not been proved. We propose a two time-scale update rule (TTUR) for training GANs with stochastic gradient descent on arbitrary GAN loss functions. TTUR has an individual learning rate for both the discriminator and the generator. Using the theory of stochastic approximation, we prove that the TTUR converges under mild assumptions to a stationary local Nash equilibrium. The convergence carries over to the popular Adam optimization, for which we prove that it follows the dynamics of a heavy ball with friction and thus prefers flat minima in the objective landscape. For the evaluation of the performance of GANs at image generation, we introduce the "Fréchet Inception Distance" (FID) which captures the similarity of generated images to real ones better than the Inception Score. In experiments, TTUR improves learning for DCGANs and Improved Wasserstein GANs (WGAN-GP) outperforming conventional GAN training on CelebA, CIFAR-10, SVHN, LSUN Bedrooms, and the One Billion Word Benchmark.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1706.08500

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 50

  1. [1][1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. J. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Józefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. G. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. A. Tucker, V. Vanhoucke, V. Vasudevan, F. B. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv e-prints , arXiv:1603.04467, 2016. - · arXiv
  2. [2][2] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. Tensorflow: A system for largescale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) , pages 265-283, 2016. -
  3. [3][3] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein GAN. arXiv e-prints , arXiv:1701.07875, 2017. - · arXiv
  4. [4][4] S. Arora, R. Ge, Y. Liang, T. Ma, and Y. Zhang. Generalization and equilibrium in generative adversarial nets (GANs). In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning , Proceedings of Machine Learning Research, vol. 70, pages 224-232, 2017. -
  5. [5][5] H. Attouch, X. Goudou, and P. Redont. The heavy ball with friction method, I. the continuous dynamical system: Global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system. Communications in Contemporary Mathematics , 2(1):1-34, 2000. -
  6. [6][6] D. Berthelot, T. Schumm, and L. Metz. BEGAN: Boundary equilibrium generative adversarial networks. arXiv e-prints , arXiv:1703.10717, 2017. - · arXiv
  7. [7][7] D. P. Bertsekas and J. N. Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM Journal on Optimization , 10(3):627-642, 2000. -
  8. [8][8] S. Bhatnagar, H. L. Prasad, and L. A. Prashanth. Stochastic Recursive Algorithms for Optimization . Lecture Notes in Control and Information Sciences. Springer-Verlag London, 2013. -
  9. [9][9] V. S. Borkar. Stochastic approximation with two time scales. Systems & Control Letters , 29(5):291-294, 1997. -
  10. [10][10] V. S. Borkar and S. P. Meyn. The O.D.E. method for convergence of stochastic approximation and reinforcement learning. SIAM Journal on Control and Optimization , 38(2):447-469, 2000. -
  11. [11][11] T. Che, Y. Li, A. P. Jacob, Y. Bengio, and W. Li. Mode regularized generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2017. arXiv:1612.02136. - · arXiv
  12. [12][12] C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv e-prints , arXiv:1312.3005, 2013. - · arXiv
  13. [13][13] D.-A. Clevert, T. Unterthiner, and S. Hochreiter. Fast and accurate deep network learning by exponential linear units (ELUs). In Proceedings of the International Conference on Learning Representations (ICLR) , 2016. arXiv:1511.07289. - · arXiv
  14. [15][15] D. C. Dowson and B. V. Landau. The Fréchet distance between multivariate normal distributions. Journal of Multivariate Analysis , 12:450-455, 1982. -
  15. [17][17] S. Gadat, F. Panloup, and S. Saadane. Stochastic heavy ball. arXiv e-prints , arXiv:1609.04228, 2016. - · arXiv
  16. [18][18] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27 , pages 2672-2680, 2014. -
  17. [19][19] I. J. Goodfellow. On distinguishability criteria for estimating generative models. In Workshop at the International Conference on Learning Representations (ICLR) , 2015. arXiv:1412.6515. - · arXiv
  18. [20][20] I. J. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. arXiv e-prints , arXiv:1701.00160, 2017. - · arXiv
  19. [21][21] X. Goudou and J. Munier. The gradient and heavy ball with friction dynamical systems: the quasiconvex case. Mathematical Programming , 116(1):173-191, 2009. -
  20. [22][22] P. Grnarova, K. Y. Levy, A. Lucchi, T. Hofmann, and A. Krause. An online learning approach to generative adversarial networks. arXiv e-prints , arXiv:1706.03269, 2017. - · arXiv
  21. [23][23] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville. Improved training of Wasserstein GANs. arXiv e-prints , arXiv:1704.00028, 2017. Advances in Neural Information Processing Systems 31 (NIPS 2017). - · arXiv
  22. [24][24] M. W. Hirsch. Convergent activation dynamics in continuous time networks. Neural Networks , 2(5):331-349, 1989. -
  23. [25][25] R. D. Hjelm, A. P. Jacob, T. Che, K. Cho, and Y. Bengio. Boundary-seeking generative adversarial networks. arXiv e-prints , arXiv:1702.08431, 2017. - · arXiv
  24. [26][26] S. Hochreiter and J. Schmidhuber. Flat minima. Neural Computation , 9(1):1-42, 1997. -
  25. [29][29] D. P. Kingma and J. L. Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR)) , 2015. arXiv:1412.6980. - · arXiv
  26. [30][30] V. R. Konda. Actor-Critic Algorithms . PhD thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, 2002. -
  27. [31][31] V. R. Konda and V. S. Borkar. Actor-critic-type learning algorithms for Markov decision processes. SIAM J. Control Optim. , 38(1):94-123, 1999. -
  28. [32][32] V. R. Konda and J. N. Tsitsiklis. Linear stochastic approximation driven by slowly varying Markov chains. Systems & Control Letters , 50(2):95-102, 2003. -
  29. [33][33] V. R. Konda and J. N. Tsitsiklis. Convergence rate of linear two-time-scale stochastic approximation. The Annals of Applied Probability , 14(2):796-819, 2004. -
  30. [34][34] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems , pages 1097-1105, 2012. -
  31. [35][35] H. J. Kushner and G. G. Yin. Stochastic Approximation Algorithms and Recursive Algorithms and Applications . Springer-Verlag New York, second edition, 2003. -
  32. [36][36] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. P. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. Photo-realistic single image super-resolution using a generative adversarial network. arXiv e-prints , arXiv:1609.04802, 2016. - · arXiv
  33. [37][37] C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, and B. Póczos. MMD GAN: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems 31 (NIPS 2017) , 2017. arXiv:1705.08584. - · arXiv
  34. [38][38] J. Li, A. Madry, J. Peebles, and L. Schmidt. Towards understanding the dynamics of generative adversarial networks. arXiv e-prints , arXiv:1706.09884, 2017. - · arXiv
  35. [39][39] J. H. Lim and J. C. Ye. Geometric GAN. arXiv e-prints , arXiv:1705.02894, 2017. - · arXiv
  36. [40][40] S. Liu, O. Bousquet, and K. Chaudhuri. Approximation and convergence properties of generative adversarial learning. In Advances in Neural Information Processing Systems 31 (NIPS 2017) , 2017. arXiv:1705.08991. - · arXiv
  37. [41][41] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , 2015. -
  38. [42][42] L. M. Mescheder, S. Nowozin, and A. Geiger. The numerics of GANs. In Advances in Neural Information Processing Systems 31 (NIPS 2017) , 2017. arXiv:1705.10461. - · arXiv
  39. [43][43] L. Metz, B. Poole, D. Pfau, and J. Sohl-Dickstein. Unrolled generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2017. arXiv:1611.02163. - · arXiv
  40. [44][44] A. Mokkadem and M. Pelletier. Convergence rate and averaging of nonlinear two-time-scale stochastic approximation algorithms. The Annals of Applied Probability , 16(3):1671-1702, 2006. -
  41. [45][45] Y. Mroueh and T. Sercu. Fisher GAN. In Advances in Neural Information Processing Systems 31 (NIPS 2017) , 2017. arXiv:1705.09675. - · arXiv
  42. [46][46] V. Nagarajan and J. Z. Kolter. Gradient descent GAN optimization is locally stable. arXiv e-prints , arXiv:1706.04156, 2017. Advances in Neural Information Processing Systems 31 (NIPS 2017). - · arXiv
  43. [48][48] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 , 2011. -
  44. [49][49] B. T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964. -
  45. [50][50] H. L. Prasad, L. A. Prashanth, and S. Bhatnagar. Two-timescale algorithms for learning Nash equilibria in general-sum stochastic games. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS '15) , pages 1371-1379, 2015. -
  46. [51][51] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2016. arXiv:1511.06434. - · arXiv
  47. [52][52] A. Ramaswamy and S. Bhatnagar. Stochastic recursive inclusion in two timescales with an application to the lagrangian dual problem. Stochastics , 88(8):1173-1187, 2016. -
  48. [53][53] T. Salimans, I. J. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training GANs. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29 , pages 2234-2242, 2016. -
  49. [54][54] V. B. Tadi´ c. Almost sure convergence of two time-scale stochastic approximation algorithms. In Proceedings of the 2004 American Control Conference , volume 4, pages 3802-3807, 2004. -
  50. [55][55] L. Theis, A. van den Oord, and M. Bethge. A note on the evaluation of generative models. In Proceedings of the International Conference on Learning Representations (ICLR) , 2016. arXiv:1511.01844. · arXiv

Cited by

Sorted by citations

Items 501–600 of 1,030

Related

See all

Metadata

Authors
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S.
Affiliations
Johannes Kepler University of Linz (AT)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2017
Volume
30
Pages
6626-6637
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1706.08500
arXiv
1706.08500
OpenAlex
W2963981733
Cited by
8,365
References
0
Citations by year
2026: 39 · 2025: 963 · 2024: 1682 · 2023: 1499 · 2022: 974 · 2021: 1505 · 2020: 1026 · 2019: 530 · 2018: 139 · 2017: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Bayer (09/2017)
  • Nvidia
  • Austrian Science Fund
  • Horizon 2020 Framework Programme (671555)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
generative adversarial networks, stochastic approximation, Adam optimizer, flat minima, Fréchet Inception Distance, Wasserstein GAN, DCGAN, image generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review