CoChat
Sign Up
PreprintCited by 20,067Gaussian Processes and Bayesian InferenceTopic ModelingMarkov Chains and Monte Carlo Methods

Auto-Encoding Variational Bayes

Open access
Kingma, Diederik P, Welling, Max·2013·arXiv
DOI 10.48550/arxiv.1312.6114arXiv 1312.6114
Open this paper in CoChat — free account

Abstract

How can we perform efficient inference and learning in directed probabilistic models, in the presence of continuous latent variables with intractable posterior distributions, and large datasets? We introduce a stochastic variational inference and learning algorithm that scales to large datasets and, under some mild differentiability conditions, even works in the intractable case. Our contributions are two-fold. First, we show that a reparameterization of the variational lower bound yields a lower bound estimator that can be straightforwardly optimized using standard stochastic gradient methods. Second, we show that for i.i.d. datasets with continuous latent variables per datapoint, posterior inference can be made especially efficient by fitting an approximate inference model (also called a recognition model) to the intractable posterior using the proposed lower bound estimator. Theoretical advantages are reflected in experimental results.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1312.6114

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Displaced vacuum states have Gaussian distributions in phase space, so the hybrid autoencoder network has a resemblance to a variational autoencoder.

    “In fact, displaced vacuum states have Gaussian distributions in phase space, so the network has a resemblance to a variational autoencoder [121].”

    From Continuous-variable quantum neural networks (2019) · cites this paper as [121]

    Section 3 (p. 5) describes a variational autoencoder with a neural-network encoder and Gaussian latent distributions, supporting the basis for the comparison. The paper does not discuss displaced vacuum states, their phase-space distributions, or the citing paper’s network, so it cannot establish the stated resemblance. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • A large amount of machine learning research is devoted to circumventing the intractable marginal likelihood problem by maximizing a lower bound called the evidence lower bound (ELBO).

    “A large amount of machine learning research is devoted to circumventing this problem by instead maximizing a suitable lower bound on this quantity, referred to as the evidence lower bound (ELBO) and sometimes as negative variational free energy.7”

    From The overfitted brain hypothesis (2021) · cites this paper as [7]

    Section 2.1 (p. 2) describes the marginal likelihood as intractable, and Section 2.2 (p. 3) describes optimizing a variational lower bound on it. That supports the sentence’s central point. Our copy does not establish the additional claim that this bound is also called “negative variational free energy.” (LLM, conf=0.79)

    Experimental — accuracy not yet measured

  • ACT is trained as a conditional VAE to capture variability in human data.

    “We implement action chunking policy with Transformers [69], an architecture designed for sequence modeling, and train it as a conditional VAE (CVAE) [59, 34] to capture the variability in human data.”

    From Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware (2023) · cites this paper as [34]

    Section 3 (p. 5) describes a variational auto-encoder trained with the AEVB algorithm, supporting the VAE component of the cited method. It does not describe a conditional VAE or its use to capture variability in human data. The sentence presents ACT’s training as the citing authors’ implementation, not as an experiment in this source. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The forward process variances β_t can be learned by reparameterization or held constant as hyperparameters.

    “The forward process variances β t can be learned by reparameterization [33] or held constant as hyperparameters”

    From Denoising Diffusion Probabilistic Models (2020) · cites this paper as [33]

    Section 2.4 (p. 4) explains how reparameterization makes an estimate differentiable with respect to distribution parameters, supporting the general learning method. The paper does not address forward-process variances β_t or the choice to hold them constant as hyperparameters, so it does not establish that specific claim. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Variational Autoencoders (VAEs) are effective methods to model the posterior.

    “Variational Autoencoders (VAEs) [99] are effective methods to model the posterior.”

    From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [99]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 16

  1. [BCV13] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. 2013.
  2. [BJP12] David M Blei, Michael I Jordan, and John W Paisley. Variational Bayesian inference with Stochastic Search. In Proceedings of the 29th International Conference on Machine Learning (ICML-12) , pages 1367-1374, 2012.
  3. [BTL13] Yoshua Bengio and ´ Eric Thibodeau-Laufer. Deep generative stochastic networks trainable by backprop. arXiv preprint arXiv:1306.1091 , 2013. · arXiv
  4. [Dev86] Luc Devroye. Sample-based non-uniform random variate generation. In Proceedings of the 18th conference on Winter simulation , pages 260-265. ACM, 1986.
  5. [DHS10] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12:21212159, 2010.
  6. [DKPR87] Simon Duane, Anthony D Kennedy, Brian J Pendleton, and Duncan Roweth. Hybrid monte carlo. Physics letters B , 195(2):216-222, 1987.
  7. [GMW13] Karol Gregor, Andriy Mnih, and Daan Wierstra. Deep autoregressive networks. arXiv preprint arXiv:1310.8499 , 2013. · arXiv
  8. [HBWP13] Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. The Journal of Machine Learning Research , 14(1):1303-1347, 2013.
  9. [HDFN95] Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal. The' wakesleep' algorithm for unsupervised neural networks. SCIENCE , pages 1158-1158, 1995.
  10. [Lin89] Ralph Linsker. An application of the principle of maximum information preservation to linear systems . Morgan Kaufmann Publishers Inc., 1989.
  11. [RGB13] Rajesh Ranganath, Sean Gerrish, and David M Blei. Black Box Variational Inference. arXiv preprint arXiv:1401.0118 , 2013. · arXiv
  12. [RMW14] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and variational inference in deep latent gaussian models. arXiv preprint arXiv:1401.4082 , 2014. · arXiv
  13. [Row98] Sam Roweis. EM algorithms for PCA and SPCA. Advances in neural information processing systems , pages 626-632, 1998.
  14. [SK13] Tim Salimans and David A Knowles. Fixed-form variational posterior approximation through stochastic linear regression. Bayesian Analysis , 8(4), 2013.
  15. [SL10] Ruslan Salakhutdinov and Hugo Larochelle. Efficient learning of deep boltzmann machines. In International Conference on Artificial Intelligence and Statistics , pages 693700, 2010.
  16. [VLL + 10] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research , 9999:3371-3408, 2010.

Cited by

Sorted by citations

Authors
Kingma, D. P., & Welling, M.
Affiliations
University of Amsterdam (NL)
ORCID
Welling, Max
Venue
arXiv
Publisher
arXiv
Published
2013
DOI
10.48550/arxiv.1312.6114
arXiv
1312.6114
OpenAlex
W1959608418
Cited by
20,067
References
0
Citations by year
2026: 414 · 2025: 1878 · 2024: 2998 · 2023: 2873 · 2022: 2374 · 2021: 3248 · 2020: 2621 · 2019: 1977 · 2018: 962 · 2017: 461 · 2016: 160 · 2015: 74 · 2014: 19 · 2013: 1 · 2012: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • UvA-DARE (University of Amsterdam) (publishedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion)
  • UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Mathematics › Statistics and Probability › Markov Chains and Monte Carlo Methods
Keywords
autoencoder, variational inference, stochastic variational inference, variational lower bound, reparameterization, continuous latent variables, recognition model, large-scale machine learning, stochastic gradient optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review