An Introduction to Variational Autoencoders

May combine two claims.

IAF greatly improves the flexibility of posterior distributions and allows for a much better fit between the posteriors and the prior.

“IAF greatly improves the flexibility of the posterior distributions, and allows for a much better fit between the posteriors and the prior.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A more flexible way to parameterize conditional distributions in directed graphical models is with neural networks.

“As we explained above, a more flexible way to parameterize such conditional distributions is with neural networks.”

May combine two claims.

VAEs, like other likelihood-based models, generate more dispersed samples but are better density models in terms of the likelihood criterion.

“VAEs, like other likelihood-based models, generate more dispersed samples, but are better density models in terms of the likelihood criterion.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Under the i.i.d. assumption, the probability of the datapoints given the parameters factorizes as a product of individual datapoint probabilities.

“Under the i.i.d.assumption, the probability of the datapoints given the parameters factorizes as a product of individual datapoint probabilities.”

• AssumptionThe authors state this as an assumption: The datapoints in dataset D are independent samples from an unchanging underlying distribution.

Probably not a substantial claim: may not pass S3 (load-bearing).

Stochastic gradient descent is a more efficient method for optimization than batch gradient descent.

“A more efficient method for optimization is stochastic gradient descent (SGD) (section A.3), which uses randomly drawn minibatches of data M ⊂ D of size N M .”

• cites section A.3Not checked yet

  Not checked in the free run.

May combine two claims.

The VAE recognition model can be arbitrarily complex but is still reasonably fast because it can be done using a single feedforward pass from input to latent variables.

“This recognition model can be arbitrary complex but is still reasonably fast because by construction it can be done using a single feedforward pass from input to latent variables.”

May combine two claims.

In ordinary variational inference, each data-case has a separate variational distribution, which is inefficient for large data-sets.

“This in contrast to VI where each data-case has a separate variational distribution, which is inefficient for large data-sets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Deep learning has been shown to work well for a large variety of classification and regression problems.

“At the time of writing, deep learning has been shown to work well for a large variety of classification and regression problems, as summarized in (LeCun et al., 2015;Goodfellow et al., 2016).”

• cites LeCun et al., 2015Not checked yet

  Not checked in the free run.

• cites Goodfellow et al., 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

If all variables in a directed graphical model are observed in the data, we can compute and differentiate the log-probability of the data under the model, leading to relatively straightforward optimization.

“If all variables in the directed graphical model are observed in the data, then we can compute and differentiate the log-probability of the data under the model, leading to relatively straightforward optimization.”

May combine two claims.

The unbiased estimator of the log-probability on a minibatch is differentiable, yielding unbiased stochastic gradients.

“The unbiased estimator log p θ (M) is differentiable, yielding the unbiased stochastic gradients:”

May combine two claims.

GANs can generate images of high subjective perceptual quality but tend to lack full support over the data.

“while GANs can generate images of high subjective perceptual quality, they tend to lack full support over the data (Grover et al., 2018), as opposed to likelihood-based generative models.”

• cites Grover et al., 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Maximization of the log-likelihood criterion is equivalent to minimization of a Kullback-Leibler divergence between the data and model distributions.

“As we will explain, maximization of the log-likelihood criterion is equivalent to minimization of a Kullback Leibler divergence between the data and model distributions.”

May combine two claims.

The Kullback-Leibler divergence between qφ(z|x) and pθ(z|x) is non-negative and zero if and only if qφ(z|x) equals the true posterior distribution.

“The second term in eq.(2.8) is the Kullback-Leibler (KL) divergence between q φ (z|x) and p θ (z|x), which is non-negative:”

• Also stated in: § 2.2 Evidence Lower Bound (ELBO) · plain

  “and zero if, and only if, q φ (z|x) equals the true posterior distribution.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The inference model in a VAE can be almost any directed graphical model.

“Like a DLVM, the inference model can be (almost) any directed graphical model:”

Probably not a substantial claim: may not pass S3 (load-bearing).

The distribution qφ(z|x) can be parameterized using deep neural networks.

“And also similar to a DLVM, the distribution q φ (z|x) can be parameterized using deep neural networks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For continuous latent variables z, the marginal pθ(x) can be seen as an infinite mixture, which is potentially more powerful than discrete mixtures.

“For continuous z, p θ (x) can be seen as an infinite mixture, which are potentially more powerful than discrete mixtures.”

May combine two claims.

Traditional inference methods are relatively expensive, often requiring a per-datapoint optimization loop or yielding bad posterior approximations.

“Traditional inference methods are relatively expensive.Such methods, for example, often require a per-datapoint optimization loop, or yield bad posterior approximations.”

May combine two claims.

The KL divergence determines the gap between the ELBO and the marginal likelihood (the tightness of the bound); the better qφ(z|x) approximates the true posterior, the smaller the gap.

“The gap between the ELBO L θ,φ (x) and the marginal likelihood log p θ (x); this is also called the tightness of the bound.The better q φ (z|x) approximates the true (posterior) distribution p θ (z|x), in terms of the KL divergence, the smaller the gap.”

Probably not a substantial claim: may not pass S3 (load-bearing).

If the latent variable z is discrete and the conditional pθ(x|z) is Gaussian, then the marginal pθ(x) is a mixture-of-Gaussians distribution.

“If z is discrete and p θ (x|z) is a Gaussian distribution, then p θ (x) is a mixtureof-Gaussians distribution.”

May combine two claims.

With amortized inference we can avoid a per-datapoint optimization loop and leverage the efficiency of stochastic gradient descent.

“With amortized inference we can avoid a per-datapoint optimization loop, and leverage the efficiency of SGD.”

May combine two claims.

Since the joint distribution pθ(x,z) is tractable to compute, a tractable marginal likelihood pθ(x) leads to a tractable posterior pθ(z|x), and vice versa; both are intractable in deep latent-variable models.

“Since p θ (x, z) is tractable to compute, a tractable marginal likelihood p θ (x) leads to a tractable posterior p θ (z|x), and vice versa.Both are intractable in DLVMs.”

May combine two claims.

Maximization of the ELBO with respect to θ and φ will concurrently approximately maximize the marginal likelihood and minimize the KL divergence of qφ(z|x) from the true posterior.

“By looking at equation 2.11, it can be understood that maximization of the ELBO L θ,φ (x) w.r.t. the parameters θ and φ, will concurrently optimize the two things we care about: 1.It will approximately maximize the marginal likelihood p θ (x).”

• Also stated in: § 2.2.1 Two for One · plain

  “2. It will minimize the KL divergence of the approximation q φ (z|x) from the true posterior p θ (z|x), so q φ (z|x) becomes better.”

Classified as probably not a substantial claim.

The optimization objective of the variational autoencoder is the evidence lower bound (ELBO).

“The optimization objective of the variational autoencoder, like in other variational methods, is the evidence lower bound, abbreviated as ELBO.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The posterior over the parameters of neural networks, p(θ|D), is generally intractable to compute exactly and requires approximate inference techniques.

“Likewise, the posterior over the parameters of (directed models parameterized with) neural networks, p(θ|D), is generally intractable to compute exactly, and requires approximate inference techniques.”

May combine two claims.

The individual-datapoint ELBO and its gradient are, in general, intractable, but good unbiased estimators exist such that minibatch SGD can still be performed.

“The individual-datapoint ELBO, and its gradient ∇ θ,φ L θ,φ (x) is, in general, intractable.However, good unbiased estimators ∇θ,φ L θ,φ (x) exist, as we will show, such that we can still perform minibatch SGD.”

May combine two claims.

We can build very flexible transformations g() for which the log-determinant is simple to compute, resulting in highly flexible inference models qφ(z|x).

“As we will show, we can build very flexible transformations g() for which log d φ (x, ) is simple to compute, resulting in highly flexible inference models q φ (z|x).”

May combine two claims.

At the start of training, the likelihood term log p(x|z) is relatively weak, so an initially attractive state is q(z|x) ≈ p(z), resulting in a stable equilibrium from which it is difficult to escape.

“At the start of training, the likelihood term log p(x|z) is relatively weak, such that an initially attractive state is where q(z|x) ≈ p(z), resulting in a stable equilibrium from which it is difficult to escape.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The ELBO can be viewed as a maximum likelihood objective in an augmented space.

“One additional perspective is that the ELBO can be viewed as a maximum likelihood objective in an augmented space.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Importance weighted estimates have notoriously bad scaling properties to high-dimensional latent spaces.

“The downside of these approaches for optimizing a tighter bound, is that importance weighted estimates have notoriously bad scaling properties to high-dimensional latent spaces.”

May combine two claims.

Issues with 'blurriness' can be countered by choosing a sufficiently flexible inference model and/or a sufficiently flexible generative model.

“Issues with 'blurriness' can thus can be countered by choosing a sufficiently flexible inference model, and/or a sufficiently flexible generative model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The free bits method is a modification of the ELBO objective that ensures that on average a certain minimum number of bits of information are encoded per latent variable, or per group of latent variables.

“An alternative proposed in (Kingma et al., 2016) is the method of free bits: a modification of the ELBO objective, that ensures that on average, a certain minimum number of bits of information are encoded per latent variable, or per group of latent variables.”

• cites Kingma et al., 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The VAE objective contains a regularization term dictated by the variational bound, lacking the usual nuisance regularization hyper-parameter required to learn useful representations.

“The VAE objective contains a regularization term dictated by the variational bound, lacking the usual nuisance regularization hyper-parameter required to learn useful representations.”

May combine two claims.

The linear autoencoder approach is limited to linear-Gaussian models, while the VAE approach applies to a much broader class of continuous latent variable models.

“However, this approach using linear autoencoders is limited to linear-Gaussian models, while our approach applies to a much broader class of continuous latent variable models.”

Probably not a substantial claim: may not pass S4 (based).

Increasing the flexibility and accuracy of the inference model generally improves the tightness of the variational bound (ELBO), bringing it closer to the true marginal likelihood objective.

“Increasing the flexibility and accuracy of the inference model wel generally improve the tightness of the variational bound (ELBO), bringing it closer the true marginal likelihood objective.”

May combine two claims.

For efficient optimization of the ELBO, the inference model must be computationally efficient to compute and differentiate its probability density qφ(z|x) and computationally efficient to sample from.

“Requirements for the inference model, in order to be able to efficiently optimize the ELBO, are that it is (1) computationally efficient to compute and differentiate its probability density q φ (z|x), and (2) computationally efficient to sample from, since both these operations need to be performed for each datapoint in a minibatch at every iteration of optimization.”

May combine two claims.

The larger variance of pθ under imperfect fit is due to the direction of the KL divergence; the reverse is not true: the generative model is only slightly penalized when putting probability mass on values of (x,z) with no support under q_D,φ.

“This is due to the direction of the KL divergence; if there are values of (x, z) which are likely under q D,φ but not under p θ , the term E q D,φ (x,z) [log p θ (x, z)] will go to infinity.However, the reverse is not true: the generative model is only slightly penalized when putting probability mass on values of (x, z) with no support under q D,φ .”

Probably not a substantial claim: may not pass S3 (load-bearing).

Planar and radial flows are effective for problems with a relatively low-dimensional latent space.

“These flows are shown to be effective to problems with a relatively low-dimensional latent space.”

• cites Rezende and Mohamed, 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Inverse autoregressive flow (IAF) is a member of the family of normalizing flows.

“As we explained, inverse autoregressive flow (IAF) is a member of the family of normalizing flows, first discussed in (Rezende and Mohamed, 2015) in the context of stochastic variational inference.”

• cites Rezende and Mohamed, 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

It is not clear how to scale planar and radial flows to much higher-dimensional latent spaces, such as latent spaces of generative models of larger images, and how they can leverage the topology of latent space as IAF can.

“It is not clear, however, how to scale such flows to much higher-dimensional latent spaces, such as latent spaces of generative models of larger images, and how planar and radial flows can leverage the topology of latent space, as is possible with IAF.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Both x and z can be composed of multiple variables with some topological ordering.

“Both x and z can be composed of multiple variables with some topological ordering.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A special case of IAF with one step and a linear autoregressive model is the fully Gaussian posterior, transforming a Gaussian variable with diagonal covariance to a Gaussian distribution with full covariance.

“It is straightforward to see that a special case of IAF with one step, and a linear autoregressive model, is the fully Gaussian posterior discussed earlier.This transforms a Gaussian variable with diagonal covariance, to one with linear dependencies, i.e. a Gaussian distribution with full covariance.”

May combine two claims.

The scalar value log pθ(x,z) - log qφ(z|x) is differentiable and optimizable with SGD.

“It is also differentiable and optimizable with SGD.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Hamiltonian Variational Inference transformations are guided by the exact posterior distribution and leave this distribution invariant for small step sizes.

“This type of transformation has the additional benefit that it is guided by the exact posterior distribution, and that it leaves this distribution invariant for small step sizes.”

• cites Salimans et al., 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The generative model pθ(x,z) and corresponding inference model qφ(z|x) can be parameterized as any directed graph.

“The generative model p θ (x, z), and corresponding inference model q φ (z|x) can be parameterized as any directed graph.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The best models to date, in terms of attained log-likelihood on test data, employ a combination of latent variables, full autoregression, and invertible transformations with tractable Jacobian determinant.

“The best models to date, in terms of attained log-likelihood on test data, employ a combination of the three approaches listed above.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the case of continuous data, autoregressive models can be interpreted as a special case of learning an invertible transformation with tractable Jacobian determinant from the data to a simpler known distribution.

“In case of continuous data, autoregressive models can be interpreted as a special case of a more general approach: learning an invertible transformation with tractable Jacobian determinant (also called normalizing flow) from the data to a simpler, known distribution such as a Gaussian or Uniform distribution.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The length of computation of the log-likelihood of fully autoregressive models does not necessarily scale with the dimensionality of the data.

“The length of computation of the log-likelihood of fully autoregressive models does not necesarilly scale with the dimensionality of the data.”

May combine two claims.

Shared ordering allows sharing parameters between inference and generative models, leading to faster learning and better solutions.

“One advantage of shared ordering, as explained in these works, is that this allows us to easily share parameters between the inference and generative models, leading to faster learning and better solutions.”

• cites Salimans, 2016Not checked yet

  Not checked in the free run.

• cites Sønderby et al., 2016aNot checked yet

  Not checked in the free run.

• cites Kingma et al., 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

When pθ(zi|Pa(zi)) is Gaussian distributed, the inference model can be directly specified as the product of this Gaussian distribution with a learned quadratic pseudo-likelihood term.

“This coupling is especially straightforward when p θ (z i |P a(z i )) is Gaussian distributed.The inference model can be directly specified as the product of this Gaussian distribution, with a learned quadratic pseudo-likelihood term:”

Probably not a substantial claim: may not pass S3 (load-bearing).

A less constraining variant is to let the neural network parameterizing qφ(zi|Pa(zi),x) be partially specified by a part of the neural network parameterizing pθ(zi|Pa(zi)).

“A less constraining variant, explored by (Kingma et al., 2016), is to simply let the neural network that parameterizes q φ (z i |P a(z i ), x) be partially specified by a part of the neural network that parameterizes p θ (z i |P a(z i )).”

• cites Kingma et al., 2016Not checked yet

  Not checked in the free run.

May combine two claims.

In Gómez-Bombarelli et al., 2018, a VAE is trained on hundreds of thousands of existing chemical structures, and the resulting continuous representation is used for gradient-based optimization towards certain properties, demonstrated on the design of drug-like molecules and organic light-emitting diodes.

“One example of a recent scientific application of artificial creativity, is shown in Gómez-Bombarelli et al., 2018.In this paper, a fairly straightforward VAE is trained on hundreds of thousands of existing chemical structures.The resulting continuous representation (latent space) is subsequently used to perform gradient-based optimization towards certain properties; the method is demonstrated on”

• cites Gómez-Bombarelli et al., 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

When no single natural one-dimensional ordering exists, like in two-dimensional images, this leads to a model with a somewhat awkward inductive bias.

“When no single natural one-dimensional ordering exists, like in two-dimensional images, this leads to a model with a somewhat awkward inductive bias.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A disadvantage of autoregressive models compared to latent-variable models is that ancestral sampling is a sequential operation of O(D) length, proportional to the dimensionality of the data.

“A disadvantage of autoregressive models, compared to latent-variable models, is that ancestral sampling from autoregressive models is a sequential operation computation of O(D) length, i.e. proportional to the dimensionality of the data.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

At the time of writing, no such model has yet been demonstrated to lead to similar performance as purely autoregressive or VAE-based models in terms of data log-likelihood.

“At the time of writing, no such model has yet been demonstrated to lead to the similar performance as purely autoregressive or VAE-based models in terms of data log-likelihood, but this remains an active area of research.”

May combine two claims.

In Bowman et al., 2015, a VAE can be successfully trained on text, and the model successfully interpolates between sentences and imputes missing words.

“A similar approach was used to generating natural-language sentences from a continuous space by Bowman et al., 2015.In this paper, it is shown how a VAE can be successfully trained on text.The model is shown to succesfully interpolate between sentences, and for imputation of missing words.”

• cites Bowman et al., 2015Not checked yet

  Not checked in the free run.

May combine two claims.

VAE-based techniques are among state of the art for semi-supervised classification, with on average under 1% classification error on MNIST when trained with only 10 labeled images per class.

“Techniques based on VAEs are now among state of the art for semisupervised classification (Maaløe et al., 2016), with on average under 1% classification error in the MNIST classification problem, when trained with only 10 labeled images per class, i.e. when more than 99.8% of the labels in the training set were removed.”

• cites Maaløe et al., 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Autoregressive models require choosing a specific ordering of input elements.

“Autoregressive models also require choosing a specific ordering of input elements (equation (4.8)).”

May combine two claims.

NICE and Real NVP flows have equal computational cost and depth in both directions, making them relatively cheap to optimize and sample from.

“In (Dinh et al., 2014;Dinh et al., 2016), particularly interesting flows (NICE and Real NVP) were introduced, with equal computational cost and depth in both directions, making it both relatively cheap to optimize and to sample from such models.”

• cites Dinh et al., 2014Not checked yet

  Not checked in the free run.

• cites Dinh et al., 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

If neural networks are used for invertible mappings, this is a powerful and flexible approach towards probabilistic modeling of continuous data and nonlinear independent component analysis.

“If we use neural networks for such invertible mappings, this is a powerful and flexible approach towards probabilistic modeling of continuous data and nonlinear independent component analysis (Deco and Brauer, 1995).”

• cites Deco and Brauer, 1995Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

At the time of writing, the best systems use a combination of autoregressive models, invertible transformations, and VAEs.

“The techniques of autoregressive models and invertible transformations can be naturally combined with variational autoencoders, and at the time of writing, the best systems use a combination Rezende and Mohamed, 2015;Kingma et al., 2016;Gulrajani et al., 2017.”

• cites Rezende and Mohamed, 2015Not checked yet

  Not checked in the free run.

• cites Kingma et al., 2016Not checked yet

  Not checked in the free run.

• cites Gulrajani et al., 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The constraint that normalizing flows iteratively update a variable of the same dimensionality as the data can make such transformations more challenging to optimize than methods without such constraint.

“This constraint on the dimensionality of intermediate states of the mapping can make such transformations more challenging to optimize than methods without such constraint.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Normalizing flows were proposed as a framework for improving the flexibility of inference models.

“Normalizing flows (Rezende and Mohamed, 2015) were proposed as a framework for improving the flexibility of inference models.”

• cites Rezende and Mohamed, 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In Kingma et al., 2016, the first normalizing flow was proposed that scales well to high-dimensional latent spaces.

“In Kingma et al., 2016, the first normalizing flow was proposed that scales well to high-dimensional latent spaces.”

• cites Kingma et al., 2016Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The ELBO objective can be generalized into an importance-weighted objective, as proposed in Burda et al., 2015.

“The ELBO objective can be generalized into an importance-weighted objective, as proposed in (Burda et al., 2015) (Importance-Weighted Autoencoders).”

• cites Burda et al., 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In Kingma et al., 2015, a refined method, the local reparameterization trick, further decreases the variance of the gradient estimator and was applied to estimation of Gaussian parameter posteriors.

“In (Kingma et al., 2015) we describe a refined method, the local reparameterization trick, for further decreasing the variance of the gradient estimator, and applied it to estimation of Gaussian parameter posteriors.”

• cites Kingma et al., 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Deep models with multiple stochastic layers can improve the variational bound, as demonstrated in Sønderby et al., 2016a and Sønderby et al., 2016b as Ladder VAEs.

“Similarly, one can use deep models with multiple stochastic layers to improve the variational bound, as demonstrated in (Sønderby et al., 2016a) and (Sønderby et al., 2016b) as Ladder VAEs.”

• cites Sønderby et al., 2016aNot checked yet

  Not checked in the free run.

• cites Sønderby et al., 2016bNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In Salimans et al., 2015, auxiliary variables were used in a combination of variational inference with Hamiltonian Monte Carlo, with momentum variables as auxiliary variables.

“In this paper, the principle was used in a combination of variational inference with Hamiltonian Monte Carlo (HMC), with the momentum variables of HMC as auxiliary variables.”

• cites Salimans et al., 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In addition to Kingma and Welling, 2014, two later papers proposed equivalent algorithms, with Lázaro-Gredilla, 2014 applying the same reparameterization gradient method to estimation of parameter posteriors rather than amortized latent-variable inference.

“In addition to our original publication (Kingma and Welling, 2014), two later papers have proposed equivalent algorithms (Rezende et al., 2014;Lázaro-Gredilla, 2014), where the latter work applies the same reparameterization gradient method to the estimation of parameter posteriors, rather than amortized latent-variable inference.”

• cites Rezende et al., 2014Not checked yet

  Not checked in the free run.

• cites Lázaro-Gredilla, 2014Not checked yet

  Not checked in the free run.

May combine two claims.

The intermediate sentences generated by the VAE are grammatically correct, and the topic and syntactic structure are typically locally consistent.

“The intermediate sentences are grammatically correct, and the topic and syntactic structure are typically locally consistent.”

• cites Bowman et al., 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Auxiliary variables can be used to improve posterior flexibility, first proposed in Salimans et al., 2015.

“As an alternative to (or in conjunction with) normalizing flows, one can use auxiliary variables to improve posterior flexibility.This principle was, to the best of our knowledge, first proposed in Salimans et al., 2015.”

• cites Salimans et al., 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Importance-weighted objective potentially reduces variance in gradient but can be difficult to scale to high-dimensional latent spaces.

“This potentially reduces the variance in the gradient, but has not been discussed in-depth here since (as often the case with importance-weighted estimators) it can be difficult to scale to highdimensional latent spaces.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Since VAEs can transform discrete observation spaces to continuous latent-variable spaces with approximately known marginals, they are interesting for use in model-based control.

“Since VAEs can transform discrete observation spaces to continuous latent-variable spaces with approximately known marginals, they are interesting for use in model-based control (Watter et al., 2015;Pritzel et al., 2017).”

• cites Watter et al., 2015Not checked yet

  Not checked in the free run.

• cites Pritzel et al., 2017Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Such models can be made incredibly flexible by parameterizing the conditional distributions with differentiable deep neural networks.

“Such models can be made incredibly flexible by parameterizing the conditional distributions with differentiable deep neural networks.”

Classified as probably not a substantial claim.

The inference model approximates the posterior distribution of the latent variables of the generative model.

“The inference model, also called encoder or recognition model, approximates the posterior distribution of the latent variables of the generative model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Both generative and inference models are directed graphical models wholly or partially parameterized by deep neural networks.

“Both the generative model and the inference model are directed graphical models that are wholly or partially parameterized by deep neural networks.”

Classified as probably not a substantial claim.

In VI, inference is cast as an optimization problem over newly introduced variational parameters, typically optimized towards the ELBO, a lower bound on model evidence or marginal likelihood.

“In VI, inference is cast as an optimization problem over newly introduced variational parameters, typically optimized towards the ELBO, a lower bound on the model evidence, or marginal likelihood of the data.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The proposed VAE framework remains the only framework in the literature that allows for both discrete and continuous observed variables, allows for efficient amortized latent-variable inference and fast synthesis, and can produce close to state-of-the-art performance in terms of log-likelihood of data.

“The proposed VAE framework remains the only framework in the literature that allows for both discrete and continuous observed variables, allows for efficient amortized latent-variable inference and fast synthesis, and which can produce close to state-of-the-art performance in terms of the log-likelihood of data.”

Classified as probably not a substantial claim.

A VAE is a combination of a deep latent-variable model with continuous latent variables and an associated inference model.

“A VAE is a combination of a deep latent-variable model (DLVM) with continuous latent variables, and an associated inference model.”

Probably not a substantial claim: may not pass S4 (based).

Stochastic gradients necessary for SGD are obtained through a basic reparameterization trick.

“Stochastic gradients, necessary for performing SGD, are obtained through a basic reparameterization trick.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The VAE framework is now a commonly used tool for various applications of probabilistic modeling and artificial creativity, and basic implementations are available in most major deep learning software libraries.

“The VAE framework is now a commonly used tool for various applications of probabilistic modeling and artificial creativity, and basic implementations are available in most major deep learning software libraries.”

Probably not a substantial claim: may not pass S4 (based).

The ELBO is a lower bound on the marginal likelihood of the data.

“The ELBO is a lower bound on the marginal likelihood of the data, also called the variational lower bound.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In Heess et al., 2015, re-parameterization of observed variables together with an observation model can be used to compute novel forms of policy gradients.

“In (Heess et al., 2015) (Stochastic Value Gradients) it was shown that the re-parameterization of the observed variables, together with an observation model, can be used to compute novel forms of policy gradients.”

• cites Heess et al., 2015Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

In latent-variable or Bayesian models with random parameters, one needs approximate posterior estimation for which variational inference methods are suitable.

“However, one is often more interested in flexible models with latent variables, such as deep latent-variable models, or Bayesian models with random parameters.In both cases one needs to perform approximate posterior estimation for which variational inference (VI) methods are suitable.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Optimization of such models towards the maximum likelihood objective is straightforward in the fully-observed case.

“Optimization of such models towards the maximum likelihood objective is straightforward in the fully-observed case.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In Higgins et al., 2017, strengthening the contribution of DKL(qφ(z|x)||pθ(z)) restricts information flow through latent space and was shown to improve disentanglement of latent factors.

“In (Higgins et al., 2017) (β-VAE) it was proposed to strengthen the contribution of D KL (q φ (z|x)||p θ (z)), thus restricting the information flow through the latent space, which was shown to improve disentanglement of latent factors, further studied in (Chen et al., 2018).”

• cites Higgins et al., 2017Not checked yet

  Not checked in the free run.

• cites Chen et al., 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Another Bayesian approach towards optimizing the likelihood pθ(x) with DLVMs is EM with MCMC.

“Another Bayesian approach towards optimizing the likelihood p θ (x) with DLVMs is Expectation Maximization (EM) with Markov Chain Monte Carlo (MCMC).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Adam and Adamax optimization methods are invariant to constant rescaling of the objective and to constant re-scalings of individual gradients.

“Adam and Adamax optimization methods for choosing α t (Kingma and Ba, 2015); these methods are invariant to constant rescaling of the objective, and invariant to constant re-scalings of the individual gradients.”

• cites Kingma and Ba, 2015Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

As a result, L(θ, ξ) only needs to be unbiased up to proportionality.

“As a result, L(θ, ξ) only needs to be unbiased up to proportionality.”

Probably not a substantial claim: may not pass S3 (load-bearing).

A principal application of Bayes' rule is that it allows us to make predictions about future data x, that are optimal as long as the prior p(θ) and model class pθ(x) are correct.

“A principal application of Bayes' rule is that it allows us to make predictions about future data x , that are optimal as long as the prior p(θ) and model class p θ (x) are correct:”

Probably not a substantial claim: may not pass S3 (load-bearing).

One version of MCMC which uses such posterior for relatively fast convergence is Hamiltonian MCMC.

“One version of MCMC which uses such posterior for relatively fast convergence, is Hamiltonian MCMC (Neal, 2011).”

• cites Neal, 2011 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

From a Bayesian perspective, we can improve upon the maximum likelihood objective through maximum a posteriori (MAP) estimation, which maximizes the log-posterior w.r.t.θ.

“From a Bayesian perspective, we can improve upon the maximum likelihood objective through maximum a posteriori (MAP) estimation, which maximizes the log-posterior w.r.t.θ.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Expectation Maximization (EM) is a general strategy for learning parameters in partially observed models.

“Expectation Maximization (EM) is a general strategy for learning parameters in partially observed models (Dempster et al., 1977).”

• cites Dempster et al., 1977Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

A disadvantage of MCMC is the requirement for running an independent MCMC chain per datapoint.

“A disadvantage of this approach is the requirement for running an independent MCMC chain per datapoint.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The tightness of this bound is defined by DKL(qφ(z|x)||pθ(z|x)).

“The tightness of this bound is defined by D KL (q φ (z|x)||p θ (z|x)).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Sharing directionality of inference (top-down) allows straightforward sharing of parameters between the generative model and the inference model.

“Sharing directionality of inference, as in (b), has the benefit that it allows for straightforward sharing of parameters between the generative model and the inference model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Due to the remarkable efficiency of reverse-mode automatic differentiation, the value and gradient of differentiable scalar objectives can be computed with equal time complexity.

“Due to the remarkable efficiency of reverse-mode automatic differentiation (also known as the backpropagation algorithm (Rumelhart et al., 1988)), the value and gradient (i.e. the vector of partial derivatives) of differentiable scalar objectives can be computed with equal time complexity.”

• cites Rumelhart et al., 1988Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Posterior gradients in DLVMs are relatively affordable to compute by differentiating the log-joint distribution w.r.t.z.

“Note that posterior gradients in DLVMs are relatively affordable to compute by differentiating the log-joint distribution w.r.t.z:”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We work with directed models where the objective per datapoint is scalar, and due to the differentiability of neural networks that compose them, the objective is differentiable w.r.t. its parameters θ.

“We work with directed models where the objective per datapoint is scalar, and due to the differentiability of neural networks that compose them, the objective is differentiable w.r.t.its parameters θ.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Early stopping criterion is to stop optimization as soon as the probability of a holdout set of data starts decreasing.

“A simple but effective criterion is to stop optimization as soon as the probability of a holdout set of data starts decreasing; this criterion is called early stopping.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Optimization of the generative model changes both pθ(zi|Pa(zi)) and pθ(x|zi, Pa(zi)).

“Optimization of the generative model changes both p θ (z i |P a(z i )) and p θ (x|z i , P a(z i )).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The prior p(θ) in equation (A.5) has diminishing effect for increasingly large N.

“The prior p(θ) in equation (A.5) has diminishing effect for increasingly large N .”

Probably not a substantial claim: may not pass S3 (load-bearing).

VAEs employ a strategy with global variational parameters.

“In contrast, VAEs employ a strategy with global variational parameters.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In case of MCMC, the posterior is approximated by a mixture of a set of approximately i.i.d. samples from the posterior, acquired by running a Markov chain.

“In case of MCMC, the posterior is approximated by a mixture of a set of approximately i.i.d.samples from the posterior, acquired by running a Markov chain.”