The DOI matches its DataCite record; 2 of the 2 references it cites were found in a registry or an index.
A note on the evaluation of generative models
Open accessAbstract
Probabilistic generative models can be used for compression, denoising, inpainting, texture synthesis, semi-supervised learning, unsupervised feature learning, and other tasks. Given this wide range of applications, it is not surprising that a lot of heterogeneity exists in the way these models are formulated, trained, and evaluated. As a consequence, direct comparison between models is often difficult. This article reviews mostly known but often underappreciated properties relating to the evaluation and interpretation of generative models with a focus on image models. In particular, we show that three of the currently most commonly used criteria---average log-likelihood, Parzen window estimates, and visual fidelity of samples---are largely independent of each other when the data is high-dimensional. Good performance with respect to one criterion therefore need not imply good performance with respect to the other criteria. Our results show that extrapolation from one criterion to another is not warranted and generative models need to be evaluated directly with respect to the application(s) they were intended for. In addition, we provide examples demonstrating that Parzen window estimates should generally be avoided.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The log likelihood bounds reported here are instead for data that has been pre-processed by adding uniform noise to remove pixel quantization, as recommended in (Theis et al., 2015).”
From Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015) · cites this paper as Theis et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1×
“…models may still be trained with respect to a different objective that is more or less related to log-likelihood, such as contrastive divergence (Hinton, 2002), score matching (Hyv¨arinen, 2005), lower bounds on the log-likelihood (Bishop, 2006), noise-contrastive estimation (Gutmann &…”
BackgroundA Parametric Texture Model Based on Joint Statistics of Complex Wavelet Coefficients
Portilla, J., & Simoncelli, E. P. (2000) · International Journal of Computer Vision · doi:10.1023/a:1026553619983 · Cited by 1,806
Flagged first, then by citations
Research works printed but not linked: 29
- Bachman, P. and Precup, D. Variational Generative Stochastic Networks with Collaborative Shap- ing. Proceedings of the 32nd International Conference on Machine Learning, pp. 1964-1972, 2015.
- Bengio, Y., Mesnil, G., Dauphin, Y., and Rifai, S. Better mixing via deep representations. In Proceedings of the 30th International Conference on Machine Learning, 2013a.
- Bengio, Y., Thibodeau-Laufer, E., Alain, G., and Yosinski, J. Deep generative stochastic networks trainable by backprop, 2013b. arXiv:1306.1091. · arXiv
- Bishop, C. M. Pattern Recognition and Machine Learning. Springer, 2006.
- Denton, E., Chintala, S., Szlam, A., and Fergus, R. Deep Generative Image Models using a Lapla- cian Pyramid of Adversarial Networks. arXiv.org, 2015.
- Dosovitskiy, A., Springenberg, J. T., and Brox, T. Learning to Generate Chairs with Convolutional Neural Networks. In IEEE International Conference on Computer Vision and Pattern Recogni- tion, 2015.
- Dziugaite, G. K., Roy, D. M., and Ghahramani, Z. Training generative neural networks via maxi- mum mean discrepancy optimization, 2015. arXiv:1505.0390.
- Gatys, L. A., Ecker, A. S., and Bethge, M. Texture synthesis and the controlled generation of natural stimuli using convolutional neural networks, 2015. arXiv:1505.07376. · arXiv
- Gerhard, H. E., Wichmann, F. A., and Bethge, M. How sensitive is the human visual system to the local statistics of natural images? PLoS Computational Biology, 9(1), 2013.
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Advances in Neural Information Processing Systems 27, 2014.
- Gregor, K., Danihelka, I., Graves, A., and Wierstra, D. DRAW: A recurrent neural network for image generation. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
- Gretton, A., Borgwardt, K. M., Rasch, M., Schölkopf, B., and Smola, A. J. A kernel method for the two-sample-problem. In Advances in Neural Information Processing Systems 20, 2007.
- Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, 2010.
- Hays, J. and Efros, A. A. Scene completion using millions of photographs. ACM Transactions on Graphics (SIGGRAPH), 26, 2007.
- Hyvärinen, A., Hurri, J., and Hoyer, P. O. Natural Image Statistics: A Probabilistic Approach to Early Computational Vision. Springer, 2009.
- Hyvärinen, A. Estimation of non-normalized statistical models using score matching. Journal of Machine Learning Research, pp. 695-709, 2005.
- Kingma, D. P., Rezende, D. J., Mohamed, S., and Welling, M. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems 27, 2014.
- Lacoste-Julien, S., Huszar, F., and Ghahramani, Z. Approximate inference for the loss-calibrated Bayesian. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, 2011.
- Lasserre, J. A., Bishop, C. M., and Minka, T. P. Principled hybrids of generative and discriminative models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2006.
- Li, Y., Swersky, K., and Zemel, R. Generative moment matching networks. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
- Shannon, C. E. A mathematical theory of communication. ACM SIGMOBILE Mobile Computing and Communications Review, 5(1):3-55, 2001.
- Sohl-Dickstein, J., Battaglino, P., and DeWeese, M. R. Minimum Probability Flow Learning. In Proceedings of the 28th International Conference on Machine Learning, 2011.
- Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, 2015.
- Theis, L. and Bethge, M. Generative Image Modeling Using Spatial LSTMs. In Advances in Neural Information Processing Systems 28, 2015.
- Uria, B., Murray, I., and Larochelle, H. RNADE: The real-valued neural autoregressive density- estimator. In Advances in Neural Information Processing Systems 26, 2013.
- van den Oord, A. and Dambre, J. Locally-connected transformations for deep GMMs, 2015. Deep Learning Workshop, ICML.
- van den Oord, A. and Schrauwen, B. Factoring Variations in Natural Images with Deep Gaussian Mixture Models. In Advances in Neural Information Processing Systems 27, 2014.
- Wang, Z. and Bovik, A. C. Mean squared error: Love it or leave it? IEEE Signal Processing Magazine, 2009.
- Welling, M. and Teh, Y. W. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In Proceedings of the 28th International Conference on Machine Learning, 2011.
Cited by
Sorted by citations
Generative adversarial networks
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2020) · Communications of the ACM · doi:10.1145/3422622 · Cited by 18,384
Generative Adversarial Networks: An Overview
Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., & Bharath, A. A. (2018) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2765202 · Cited by 4,540
Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., & Ganguli, S. (2015) · arXiv · doi:10.48550/arxiv.1503.03585 · Cited by 2,637
Tacotron: towards end-to-end speech synthesis
Wang, Y., Skerry-Ryan, R. J., Stanton, D., Wu, Y., Weiss, R. J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1452 · Cited by 1,746
Diffusion Models: A Comprehensive Survey of Methods and Applications
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., & Yang, M.-H. (2024) · ACM Computing Surveys · doi:10.1145/3626235 · Cited by 1,704
Metadata
- Authors
- Theis, L., Oord, A. V. D., & Bethge, M.
- Affiliations
- University of Tübingen (DE)
- ORCID
- Bethge, Matthias
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1511.01844
- OpenAlex
- W2099057450
- Cited by
- 1,026
- References
- 2
- Citations by year
- 2026: 40 · 2025: 62 · 2024: 83 · 2023: 109 · 2022: 71 · 2021: 136 · 2020: 148 · 2019: 130 · 2018: 114 · 2017: 85 · 2016: 41 · 2015: 6
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Max Planck Digital Library (submittedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Signal Denoising Methods
- Keywords
- generative model evaluation, probabilistic generative models, image models, visual fidelity, high-dimensional data, model comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
