Importance Weighted Autoencoders
Open accessAbstract
The variational autoencoder (VAE; Kingma, Welling (2014)) is a recently proposed generative model pairing a top-down generative network with a bottom-up recognition network which approximates posterior inference. It typically makes strong assumptions about posterior inference, for instance that the posterior distribution is approximately factorial, and that its parameters can be approximated with nonlinear regression from the observations. As we show empirically, the VAE objective can lead to overly simplified representations which fail to use the network's entire modeling capacity. We present the importance weighted autoencoder (IWAE), a generative model with the same architecture as the VAE, but which uses a strictly tighter log-likelihood lower bound derived from importance weighting. In the IWAE, the recognition network uses multiple samples to approximate the posterior, giving it increased flexibility to model complex posteriors which do not fit the VAE modeling assumptions. We show empirically that IWAEs learn richer latent space representations than VAEs, leading to improved test log-likelihood on density estimation benchmarks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The drop in variance can be attributed to additional un-correlated randomness from sampling more when reparametrizing for each ELBO.
“This drop in variance can be attributed to two factors: (1) additional un-correlated randomness from sampling more when reparametrizing for each ELBO [4]”
From Multimodal Generative Models for Scalable Weakly-Supervised Learning (2018) · cites this paper as [4]
Section 3.1 (p. 4) says the method draws multiple samples and uses reparameterization to obtain a low-variance update rule. Section 2 (p. 3) attributes the VAE update’s lower variance to its use of log-likelihood gradients, not to additional uncorrelated randomness. The cited paper therefore supports the sampling and low-variance elements, but not the proposed explanation for the drop in variance. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 25
- Ba, J. L., Mnih, V., and Kavukcuoglu, K. Multiple object recognition with visual attention. In International Conference on Learning Representations , 2015.
- Bornschein, J. and Bengio, Y. Reweighted wake-sleep. International Conference on Learning Representations , 2015.
- Burda, Y., Grosse, R. B., and Salakhutdinov, R. Accurate and conservative estimates of MRF log-likelihood using reverse annealing. Artificial Intelligence and Statistics , pp. 102-110, 2015.
- Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Artificial Intelligence and Statistics , pp. 249-256, 2010.
- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Neural Information Processing Systems , 2014.
- Gregor, K., Danihelka, I., Mnih, A., Blundell, C., and Wierstra, D. Deep autoregressive networks. International Conference on Machine Learning , 2014.
- Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., and Wierstra, D. DRAW: A recurrent neural network for image generation. In International Conference on Machine Learning , pp. 1462-1471, 2015.
- Grosse, R. and Salakhudinov, R. Scaling up natural gradient by sparsely factorizing the inverse fisher matrix. In International Conference on Machine Learning , 2015.
- Kingma, D. and Ba, J. L. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.
- Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. International Conference on Learning Representations , 2014.
- Kingma, D. P., Mohamed, S., Rezende, D. J., and Welling, M. Semi-supervised learning with deep generative models. In Neural Information Processing Systems , 2014.
- Kulkarni, T. D., Whitney, W., Kohli, P., and Tenenbaum, J. B. Deep convolutional inverse graphics network. arXiv:1503.03167, 2015. · arXiv
- Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. One-shot learning by inverting a compositional causal process. In Neural Information Processing Systems , 2013.
- Larochelle, H., Murray I. The neural autoregressive distribution estimator. In Artificial Intelligence and Statistics , pp. 29-37, 2011.
- LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.
- Li, Y., Swersky, K., and Zemel, R. Generative moment matching networks. In International Conference on Machine Learning , pp. 1718-1727, 2015.
- Mnih, A. and Gregor, K. Neural variational inference and learning in belief networks. In International Conference on Machine Learning , pp. 1791-1799, 2014.
- Murray, I. and Salakhutdinov, R. Evaluating probabilities under high-dimensional latent variable models. In Neural Information Processing Systems , pp. 1137-1144, 2009.
- Rezende, D. J. and Mohamed, S. Variational inference with normalizing flows. In International Conference on Machine Learning , pp. 1530-1538, 2015.
- Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. International Conference on Machine Learning , pp. 1278-1286, 2014.
- Salakhutdinov, R. and E., Hinton G. Deep Boltzmann machines. In Neural Information Processing Systems , 2009.
- Salakhutdinov, R. and Murray, I. On the quantitative analysis of deep belief networks. In International Conference on Machine Learning , 2008.
- Salimans, T., Kingma, D. P., and Welling, M. Markov chain Monte Carlo and variational inference: bridging the gap. In International Conference on Machine Learning , pp. 1218-1226, 2015.
- Smolensky, P. Information processing in dynamical systems: foundations of harmony theory. In Rumelhart, D. E. and McClelland, J. L. (eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition . MIT Press, 1986.
- Tang, Y. and Salakhutdinov, R. Learning stochastic feedforward neural networks. In Neural Information Processing Systems , 2013.
Cited by
Sorted by citations
An Introduction to Variational Autoencoders
Kingma, D. P., & Welling, M. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000056 · Cited by 2,609 · see citing works
Density estimation using Real NVP
Dinh, L., Sohl-Dickstein, J., & Bengio, S. (2016) · arXiv · Cited by 2,067 · see citing works
Multimodal Generative Models for Scalable Weakly-Supervised Learning
Wu, M., & Goodman, N. (2018) · arXiv · doi:10.48550/arxiv.1802.05335 · Cited by 200 · see citing works
Towards Deeper Understanding of Variational Autoencoding Models
Zhao, S., Song, J., & Ermon, S. (2017) · arXiv · doi:10.48550/arxiv.1702.08658 · Cited by 116 · see citing works
Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models
Shi, Y., Siddharth, N., Paige, B., & Torr, P. H. S. (2019) · arXiv · doi:10.48550/arxiv.1911.03393 · Cited by 87 · see citing works
“First, let’s take a look at ELBO: maximising the standard ELBO indirectly minimises the KL between the variational and true posteriors, since Maximising the IWAE objective however, indirectly minimises the KL between implicit posteri-ors (Le et al., 2018) as where leading to higher-entropy estimates of the variational…”
BackgroundInfluentialMultimodal Fusion Refiner Networks
Sankaran, S., Yang, D., & Lim, S.-N. (2021) · arXiv · doi:10.48550/arxiv.2104.03435 · Cited by 10 · see citing works
Sutter, T. M., Daunhawer, I., & Vogt, J. E. (2021) · arXiv · doi:10.48550/arxiv.2105.02470 · Cited by 41 · see citing works
- Authors
- Burda, Y., Grosse, R., & Salakhutdinov, R.
- Affiliations
- University of Toronto (CA)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2015
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1509.00519
- OpenAlex
- W1779483307
- Cited by
- 644
- References
- 0
- Citations by year
- 2025: 27 · 2024: 31 · 2023: 48 · 2022: 50 · 2021: 111 · 2020: 127 · 2019: 112 · 2018: 69 · 2017: 40 · 2016: 24 · 2015: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- TIB Data Manager
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- variational autoencoder, importance weighting, posterior approximation, latent representations, density estimation, generative models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
