Wasserstein Auto-Encoders
Open accessThe DOI matches its DataCite record; 38 of the 38 references it cites were found in a registry or an index.
Abstract
We propose the Wasserstein Auto-Encoder (WAE)---a new algorithm for building a generative model of the data distribution. WAE minimizes a penalized form of the Wasserstein distance between the model distribution and the target distribution, which leads to a different regularizer than the one used by the Variational Auto-Encoder (VAE). This regularizer encourages the encoded training distribution to match the prior. We compare our algorithm with several other techniques and show that it is a generalization of adversarial auto-encoders (AAE). Our experiments show that WAE shares many of the properties of VAEs (stable training, encoder-decoder architecture, nice latent manifold structure) while generating samples of better quality, as measured by the FID score.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 5×
“• Empirical evaluation of WAE on MNIST and CelebA datasets with squared cost c(x, y) = ‖x−y‖22.”
MethodInfluentialBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · cited 1×
“999, and convolutional deep neural network architectures for encoder mapping Qφ : X → Z and decoder mapping Gθ : Z → X similar to the DCGAN ones reported by [28] with batch normalization [29].”
MethodIdentity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665 · cited 1×
“The bigVAE, bigWAE-MMD, and bigWAE-GAN also included in the table are the best models based on larger scale study reported in Supplementary D, which involved a much wider hyperparameter sweep, as well as using ResNet50-v2 [33] architecture for the encoder and decoder mappings.”
MethodGANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747
Improved training of Wasserstein GANs
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017) · arXiv · doi:10.48550/arxiv.1704.00028 · Cited by 6,800 · cited 2×
“ches have been proposed in the literature to circumvent this diculty. For W 1 [4] tried to implement the constraint in the dual formulation (2) by clipping the weights of the neural network f. Later [7] proposed to relax the same constraint by penalizing the objective of (2) with a term E(krf(X)k 1)2 which should not…”
MethodTopics in Optimal Transportation
Villani, C. (2003) · Graduate Studies in Mathematics · doi:10.1090/gsm/058 · Cited by 4,412
(2012) · MPG.PuRe (Max Planck Society) · doi:10.5555/2188385.2188410 · Cited by 3,603
Self-citationSinkhorn Distances: Lightspeed Computation of Optimal TransportPreprint
Cuturi, M. (2013) · Cited by 2,385
Arjovsky, M., Chintala, S., & Bottou, L. (2017) · arXiv · doi:10.48550/arxiv.1701.07875 · Cited by 2,225
Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., & Frey, B. (2015) · arXiv · doi:10.48550/arxiv.1511.05644 · Cited by 1,579
f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization
Nowozin, S., Cseke, B., & Tomioka, R. (2016) · arXiv · doi:10.48550/arxiv.1606.00709 · Cited by 1,183 · cited 2×
“The OT cost [5] is a way to measure a distance between probability distributions and provides a much weaker topology than many others, including f -divergences associated with the original GAN algorithms [6].”
MethodEnergy-based Generative Adversarial Network
Zhao, J., Mathieu, M., & LeCun, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.03126 · Cited by 912 · cited 1×
“There have been many other approaches trying to blend the adversarial training of GANs with auto-encoder architectures [17, 18, 19, 20].”
BackgroundBEGAN: Boundary Equilibrium Generative Adversarial Networks
Berthelot, D., Schumm, T., & Metz, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.10717 · Cited by 654
Adversarially Learned Inference
Dumoulin, V., Belghazi, M. I., Poole, B., Mastropietro, O., Lamb, A., Arjovsky, M., & Courville, A. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00704 · Cited by 579
GANs Trained by a Two Time-Scale Update Rule Converge to a Nash EquilibriumPreprint
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., & Hochreiter, S. (2017) · arXiv (Cornell University) · Cited by 553
Generative Moment Matching Networks
Li, Y., Swersky, K., & Zemel, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.02761 · Cited by 517
Liero, M., Mielke, A., & Savaré, G. (2017) · Inventiones mathematicae · doi:10.1007/s00222-017-0759-8 · Cited by 500
Stochastic Optimization for Large-scale Optimal TransportPreprint
Genevay, A., Cuturi, M., Peyré, G., & Bach, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08527 · Cited by 491
MMD GAN: Towards Deeper Understanding of Moment Matching Network
Li, C., Chang, W.-C., Cheng, Y., Yang, Y., & Póczos, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.08584 · Cited by 384
InfoVAE: Balancing Learning and Inference in Variational Autoencoders
Zhao, S., Song, J., & Ermon, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33015885 · Cited by 265
Unbalanced optimal transport: Dynamic and Kantorovich formulations
Chizat, L., Peyré, G., Schmitzer, B., & Vialard, F.-X. (2018) · Journal of Functional Analysis · doi:10.1016/j.jfa.2018.03.008 · Cited by 263
From optimal transport to generative modeling: the VEGAN cookbookPreprint
Bousquet, O., Gelly, S., Tolstikhin, I., Simon-Gabriel, C.-J., & Schölkopf, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07642 · Cited by 165
Self-citationTraining generative neural networks via Maximum Mean Discrepancy optimization
Dziugaite, G. K., Roy, D. M., & Ghahramani, Z. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.03906 · Cited by 159
Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
Mescheder, L., Nowozin, S., & Geiger, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.04722 · Cited by 143
Liese, F., & Miescke, K. J. (2007) · Springer series in statistics · doi:10.1007/978-0-387-73194-0_3 · Cited by 101
It Takes (Only) Two: Adversarial Generator-Encoder Networks
Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11449 · Cited by 95
Improved generator objectives for GANsPreprint
Poole, B., Alemi, A. A., Sohl‐Dickstein, J., & Angelova, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.02780 · Cited by 72
On the High Dimensional Power of a Linear-Time Two Sample Test under Mean-shift Alternatives
Reddi, S. J., Ramdas, A., Póczos, B., Singh, A., & Wasserman, L. (2015) · International Conference on Artificial Intelligence and Statistics · Cited by 22
Unbalanced Optimal Transport: Dynamic and Kantorovich Formulation
Chizat, L., Peyré, G., Schmitzer, B., & Vialard, F.-X. (2015) · arXiv · doi:10.48550/arxiv.1508.05216
Liero, M., Mielke, A., & Savaré, G. (2015) · arXiv · doi:10.48550/arxiv.1508.07941
Research works printed but not linked: 2
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pages 2672-2680, 2014.
- M. D. Hoffman and M. Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In NIPS Workshop on Advances in Approximate Bayesian Inference, 2016.
Cited by
Generalizing to unseen domains: A survey on domain generalization
Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., Chen, Y., Zeng, W., & Yu, P. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3178128 · Cited by 1,046
““fictitious” yet “challenging” populations, where a Wasserstein Auto-Encoder (WAE) [176] was used to help generate samples that preserve the semantic and have large domain transportation.”
MethodVIBE: Video Inference for Human Body Pose and Shape Estimation
Kocabas, M., Athanasiou, N., & Black, M. J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00530 · Cited by 1,031
Machine learning–enabled high-entropy alloy discovery
Rao, Z., Tung, P.-Y., Xie, R., Wei, Y., Zhang, H., Ferrari, A., Klaver, T. P. C., Körmann, F., Sukumar, P. T., Kwiatkowski da Silva, A., et al. (2022) · Science · doi:10.1126/science.abo4940 · Cited by 745
Bond-Taylor, S., Leach, A., Long, Y., & Willcocks, C. G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3116668 · Cited by 703
DeepSMOTE: Fusing Deep Learning and SMOTE for Imbalanced Data
Dablain, D., Krawczyk, B., & Chawla, N. V. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3136503 · Cited by 591
Metadata
- Authors
- Tolstikhin, I., Bousquet, O., Gelly, S., & Schoelkopf, B.
- Affiliations
- Max Planck Institute for Intelligent Systems (DE)
- Google (United States) (US)
- ORCID
- Bousquet, Olivier
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1711.01558
- OpenAlex
- W2767757834
- Cited by
- 695
- References
- 35
- Citations by year
- 2026: 81 · 2025: 91 · 2024: 73 · 2023: 76 · 2022: 68 · 2021: 92 · 2020: 87 · 2019: 90 · 2018: 37
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- generative models, Wasserstein distance, variational autoencoder, regularization, sample quality, image generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.