The Cramer Distance as a Solution to Biased Wasserstein Gradients
Open accessAbstract
The Wasserstein probability metric has received much attention from the machine learning community. Unlike the Kullback-Leibler divergence, which strictly measures change in probability, the Wasserstein metric reflects the underlying geometry between outcomes. The value of being sensitive to this geometry has been demonstrated, among others, in ordinal regression and generative modelling. In this paper we describe three natural properties of probability divergences that reflect requirements from machine learning: sum invariance, scale sensitivity, and unbiased sample gradients. The Wasserstein metric possesses the first two properties but, unlike the Kullback-Leibler divergence, does not possess the third. We provide empirical evidence suggesting that this is a serious issue in practice. Leveraging insights from probabilistic forecasting we propose an alternative to the Wasserstein metric, the Cramér distance. We show that the Cramér distance possesses all three desired properties, combining the best of the Wasserstein and Kullback-Leibler divergences. To illustrate the relevance of the Cramér distance in practice we design a new algorithm, the Cramér Generative Adversarial Network (GAN), and show that it performs significantly better than the related Wasserstein GAN.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CramerGAN performs significantly better than WGAN.
“The CramerGANperformssignificantly better than the WGAN.”
From Countering Malicious DeepFakes: Survey, Battleground, and Horizon (2022) · cites this paper as Bellemare et al. 2017
Section 5.1 (p. 7) reports that Cramér GAN produced more stable learning and more diverse samples than WGAN-GP; the experiments on p. 8 give examples of both advantages. The claim overgeneralizes this comparison to WGAN and says “significantly better” without specifying the outcomes or conditions the paper evaluated. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 30 · Cited by 223
Research works printed but not linked: 30
- Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein GAN. arXiv preprint arXiv:1701.07875 . · arXiv
- Bellemare, M. G., Dabney, W., and Munos, R. (2017). A distributional perspective on reinforcement learning. In Proceedings of the International Conference on Machine Learning, to appear .
- Bickel, P. J. and Freedman, D. A. (1981). Some asymptotic theory for the bootstrap. The Annals of Statistics , pages 1196-1217.
- Chung, K.-J. and Sobel, M. J. (1987). Discounted MDP's: Distribution functions and exponential utility maximization. SIAM Journal on Control and Optimization , 25(1):49-62.
- Cover, T. M. and Thomas, J. A. (1991). Elements of information theory . John Wiley & Sons.
- Danihelka, I., Lakshminarayanan, B., Uria, B., Wierstra, D., and Dayan, P. (2017). Comparison of Maximum Likelihood and GAN-based training of Real NVPs. arXiv preprint arXiv:1705.05263 . · arXiv
- Dudley, R. M. (2002). Real analysis and probability , volume 74. Cambridge University Press.
- Dziugaite, G. K., Roy, D. M., and Ghahramani, Z. (2015). Training generative neural networks via maximum mean discrepancy optimization. In Proceedings of the Conference on Uncertainty in Artificial Intelligence .
- Esfahani, P. M. and Kuhn, D. (2015). Data-driven distributionally robust optimization using the Wasserstein metric: Performance guarantees and tractable reformulations. arXiv preprint arXiv:1505.05116 . · arXiv
- Frogner, C., Zhang, C., Mobahi, H., Araya, M., and Poggio, T. A. (2015). Learning with a Wasserstein loss. In Advances in Neural Information Processing Systems .
- Gao, R. and Kleywegt, A. J. (2016). Distributionally robust stochastic optimization with Wasserstein distance. arXiv preprint arXiv:1604.02199 . · arXiv
- Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association , 102(477):359-378.
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adversarial nets. In Advances in Neural Information Processing Systems .
- Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. (2012). A kernel twosample test. Journal of Machine Learning Research , 13:723-773.
- Hernández-Lobato, J. M. and Adams, R. P. (2015). Probabilistic backpropagation for scalable learning of Bayesian neural networks. In Proceedings of the International Conference on Machine Learning .
- Isola, P., Zhu, J.-Y ., Zhou, T., and Efros, A. A. (2016). Image-to-image translation with conditional adversarial networks. arXiv preprint arXiv:1611.07004 . · arXiv
- Kingma, D. P. and Welling, M. (2014). Auto-encoding variational Bayes. Proceedings of the International Conference on Learning Representations .
- Li, C.-L., Chang, W.-C., Cheng, Y., Yang, Y., and Póczos, B. (2017). MMD GAN: Towards deeper understanding of moment matching network. arXiv preprint arXiv:1705.08584 . · arXiv
- Li, Y., Swersky, K., and Zemel, R. (2015). Generative moment matching networks. In Proceedings of the International Conference on Machine Learning .
- Liu, Z., Luo, P., Wang, X., and Tang, X. (2015). Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision .
- Montavon, G., Müller, K.-R., and Cuturi, M. (2016). Wasserstein training of restricted Boltzmann machines. In Advances in Neural Information Processing Systems .
- Mroueh, Y., Sercu, T., and Goel, V. (2017). McGan: Mean and covariance feature matching GAN. arXiv preprint arXiv:1702.08398 . · arXiv
- Müller, A. (1997). Integral probability metrics and their generating classes of functions. Advances in Applied Probability , 29(2):429-443.
- Rachev, S. T., Klebanov, L., Stoyanov, S. V., and Fabozzi, F. (2013). The methods of distances in the theory of probability and statistics . Springer.
- Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 . · arXiv
- Rizzo, M. L. and Székely, G. J. (2016). Energy distance. Wiley Interdisciplinary Reviews: Computational Statistics , 8(1):27-38.
- Rubner, Y., Tomasi, C., and Guibas, L. J. (2000). The earth mover's distance as a metric for image retrieval. International journal of computer vision , 40(2):99-121.
- Van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016). Pixel recurrent neural networks. In Proceedings of the International Conference on Machine Learning .
- Veraar, M. (2010). On Khintchine inequalities with a weight. Proceedings of the American Mathematical Society , 138(11):4119-4121.
- Zolotarev, V. M. (1976). Metric distances in spaces of random variables and their distributions. Sbornik: Mathematics , 30(3):373-401.
- Authors
- Bellemare, M. G., Danihelka, I., Dabney, W., Mohamed, S., Lakshminarayanan, B., Hoyer, S., & Munos, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1705.10743
- OpenAlex
- W2619903301
- Cited by
- 223
- References
- 30
- Citations by year
- 2025: 11 · 2024: 16 · 2023: 24 · 2022: 19 · 2021: 46 · 2020: 38 · 2019: 28 · 2018: 29 · 2017: 12
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Mathematics › Applied Mathematics › Geometric Analysis and Curvature Flows
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- Wasserstein distance, unbiased gradient estimation, Kullback-Leibler divergence, scale sensitivity, Wasserstein GAN, probabilistic forecasting, generative modelling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
