Generating Sentences from a Continuous Space
Open accessAbstract
The standard recurrent neural network language model (rnnlm) generates sentences one word at a time and does not work from an explicit global sentence representation.In this work, we introduce and study an rnn-based variational autoencoder generative model that incorporates distributed latent representations of entire sentences.This factorization allows it to explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features.Samples from the prior over these sentence representations remarkably produce diverse and well-formed sentences through simple deterministic decoding.By examining paths through this latent space, we are able to generate coherent novel sentences that interpolate between known sentences.We present techniques for solving the difficult learning problem presented by this model, demonstrate its effectiveness in imputing missing words, explore many interesting properties of the model's latent sentence space, and present negative results on the use of the model in language modeling.but now , as they parked out front and owen stepped out of the car , he could see True: that the transition was complete .RNNLM: it , " i said .VAE: through the driver 's door .you kill
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Word dropout and KL annealing are crucial to prevent latent collapse.
“As in [2], word dropout and KL annealing are crucial to prevent latent collapse.”
From Multimodal Generative Models for Scalable Weakly-Supervised Learning (2018) · cites this paper as [2]
Section 3.1, “Optimization challenges,” identifies the model’s tendency to ignore the latent variable and presents KL annealing and word dropout as techniques to mitigate it. The KL cost annealing and Section 6.1 passages describe improved use of the latent variable, but the paper does not establish that both techniques are crucial or that they reliably prevent collapse. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 36
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. ICLR.
- Justin Bayer and Christian Osendorfer. 2015. Learning stochastic recurrent networks. arXiv preprint arXiv:1411.7610. · arXiv
- Julian Besag. 1986. On the statistical analysis of dirty pictures. Journal of the Royal Statistical Society Series B (Methodological) pages 48-259.
- Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. 2015. A recurrent latent variable model for sequential data. In Proc. NIPS .
- Andrew M. Dai and Quoc V. Le. 2015. Semi- supervised sequence learning. In Proc. NIPS .
- Emily Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. 2015. Deep generative image models using a laplacian pyramid of adversarial networks. In Proc. NIPS .
- Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large para- phrase corpora: Exploiting massively parallel news sources. In Proc. COLING.
- Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional net- works for visual recognition and description. In Proc. CVPR.
- Otto Fabius and Joost R. van Amersfoort. 2014. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.6581. · arXiv
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proc. NIPS .
- Karol Gregor, Ivo Danihelka, Alex Graves, and Daan Wierstra. 2015. DRAW: A recurrent neu- ral network for image generation. In Proc. ICML.
- Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A kernel two-sample test. JMLR 13(1):723-773.
- Mohit Iyyer, Jordan Boyd-Graber, and Hal Daumé III. 2014. Generating sentences from seman- tic vector space representations. In Proc. NIPS Workshop on Learning Semantics.
- Mohit Iyyer, Varun Manjunatha, Jordan Boyd- Graber, and Hal Daumé III. 2015. Deep un- ordered composition rivals syntactic methods for text classification. In Proc. ACL.
- Michael Kearns, Yishay Mansour, and Andrew Y Ng. 1998. An information-theoretic analysis of hard and soft assignment methods for clustering. In Learning in graphical models, Springer, pages 495-520.
- Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proc. EMNLP .
- Diederik P. Kingma and Max Welling. 2015. Auto- encoding variational bayes. In Proc. ICLR.
- Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S Zemel, Antonio Torralba, Raquel Ur- tasun, and Sanja Fidler. 2015. Skip-thought vec- tors. In Proc. NIPS .
- Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Pe- ter Ondruska, Ishaan Gulrajani, and Richard Socher. 2016. Ask me anything: Dynamic mem- ory networks for natural language processing. In Proc. ICML.
- Quoc V. Le and Tomáš Mikolov. 2014. Distributed representations of sentences and documents. In Proc. ICML.
- Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015a. A hierarchical neural autoencoder for paragraphs and documents. In Proc. ACL.
- Xin Li and Dan Roth. 2002. Learning question classifiers. In Proc. COLING.
- Yujia Li, Kevin Swersky, and Richard Zemel. 2015b. Generative moment matching networks. In Proc. ICML.
- Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhi- heng Huang, and Alan Yuille. 2015. Deep cap- tioning with multimodal recurrent neural net- works (m-RNN). In Proc. ICLR.
- Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large an- notated corpus of English: The Penn Treebank. Computational linguistics 19(2):313-330.
- Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neu- ral variational inference for text processing. In Proc. ICML.
- Tomáš Mikolov, Stefan Kombrink, Lukáš Burget, Jan Honza Černockỳ, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In Proc. ICASSP .
- Tapani Raiko, Mathias Berglund, Guillaume Alain, and Laurent Dinh. 2015. Techniques for learning binary stochastic feedforward neu- ral networks. In Proc. ICLR.
- Danilo J. Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In Proc. ICML.
- Danilo J. Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative mod- els. In Proc. ICML.
- Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D. Manning, and Andrew Y. Ng. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proc. NIPS .
- Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. JMLR 15(1):1929-1958.
- Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In Proc. NIPS .
- Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proc. NIPS .
- Lucas Theis and Matthias Bethge. 2015. Gener- ative image modeling using spatial LSTMs. In Proc. NIPS .
- Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. In Proc. IJCAI .
Cited by
Sorted by citations
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules
Gómez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J. M., Sánchez-Lengeling, B., Sheberla, D., Aguilera-Iparraguirre, J., Hirzel, T. D., Adams, R. P., & Aspuru-Guzik, A. (2018) · ACS Central Science · doi:10.1021/acscentsci.7b00572 · Cited by 3,106 · see citing works
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921 · see citing works
An Introduction to Variational Autoencoders
Kingma, D. P., & Welling, M. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000056 · Cited by 2,609 · see citing works
Generating Focused Molecule Libraries for Drug Discovery with Recurrent Neural Networks
Segler, M. H. S., Kogej, T., Tyrchan, C., & Waller, M. P. (2018) · ACS Central Science · doi:10.1021/acscentsci.7b00512 · Cited by 1,537 · see citing works
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · see citing works
Disease variant prediction with deep generative models of evolutionary data
Frazer, J., Notin, P., Dias, M., Gomez, A., Min, J. K., Brock, K., Gal, Y., & Marks, D. S. (2021) · Nature · doi:10.1038/s41586-021-04043-8 · Cited by 791 · see citing works
Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
Schönfeld, E., Ebrahimi, S., Sinha, S., Darrell, T., & Akata, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00844 · Cited by 747 · see citing works
Density estimation using Real NVP
Dinh, L., Sohl-Dickstein, J., & Bengio, S. (2016) · arXiv · Cited by 2,067 · see citing works
Self-citationA Survey on Dialogue Systems: Recent Advances and New Frontiers
Chen, H., Liu, X., Yin, D., & Tang, J. (2017) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3166054.3166058 · Cited by 579 · see citing works
Stock Movement Prediction from Tweets and Historical Prices
Xu, Y., & Cohen, S. B. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1183 · Cited by 434 · see citing works
Oh, C., Song, J., Choi, J., Kim, S., Lee, S., & Suh, B. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174223 · Cited by 301 · see citing works
Generating Sentences by Editing Prototypes
Guu, K., Hashimoto, T. B., Oren, Y., & Liang, P. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00030 · Cited by 300 · see citing works
“Second, our parameterization avoids collapsing the latent code, which is a serious problem with variational autoencoders in practice (Bowman et al., 2016).”
BackgroundMethodInfluentialControllable Abstractive Summarization
Fan, A., Grangier, D., & Auli, M. (2018) · Proceedings of the 2nd Workshop on Neural Machine Translation and Generation · doi:10.18653/v1/w18-2706 · Cited by 281 · see citing works
“This is exemplified by work focusing on natural language generation such as (Bowman et al., 2016; Yu et al., 2017; Zhao et al., 2017; Rajeswar et al., 2017).”
BackgroundHybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generation
Li, C. Y., Liang, X., Hu, Z., & Xing, E. P. (2018) · arXiv · doi:10.48550/arxiv.1805.08298 · Cited by 271 · see citing works
Unsupervised speech representation learning using WaveNet autoencoders
Chorowski, J., Weiss, R. J., Bengio, S., & van den Oord, A. (2019) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2019.2938863 · Cited by 325 · see citing works
Self-citationPandi, A., Adam, D., Zare, A., Trinh, V. T., Schaefer, S. L., Burt, M., Klabunde, B., Bobkova, E., Kushwaha, M., Foroughijabbari, Y., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-42434-9 · Cited by 173 · see citing works
Unifying Human and Statistical Evaluation for Natural Language Generation
Hashimoto, T. B., Zhang, H., & Liang, P. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1169 · Cited by 161 · see citing works
Gmail Smart Compose: Real-Time Assisted Writing
Chen, M. X., Lee, B. N., Bansal, G., Cao, Y., Zhang, S., Lu, J., Tsay, J., Wang, Y., Dai, A. M., Chen, Z., et al. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330723 · Cited by 169 · see citing works
Self-citationMultimodal Generative Models for Scalable Weakly-Supervised Learning
Wu, M., & Goodman, N. (2018) · arXiv · doi:10.48550/arxiv.1802.05335 · Cited by 200 · see citing works
SenseGen: A deep learning architecture for synthetic sensor data generation
Alzantot, M., Chakraborty, S., & Srivastava, M. (2017) · 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomw.2017.7917555 · Cited by 153 · see citing works
Towards Deeper Understanding of Variational Autoencoding Models
Zhao, S., Song, J., & Ermon, S. (2017) · arXiv · doi:10.48550/arxiv.1702.08658 · Cited by 116 · see citing works
On the creativity of large language models
Franceschelli, G., & Musolesi, M. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02127-3 · Cited by 113 · see citing works
Deciphering protein evolution and fitness landscapes with latent space models
Ding, X., Zou, Z., & Brooks III, C. L. (2019) · Nature Communications · doi:10.1038/s41467-019-13633-0 · Cited by 106 · see citing works
Autonomous Molecular Design: Then and Now
Dimitrov, T., Kreisbeck, C., Becker, J. S., Aspuru-Guzik, A., & Saikin, S. K. (2019) · ACS Applied Materials & Interfaces · doi:10.1021/acsami.9b01226 · Cited by 103 · see citing works
AlQuraishi, M., & Sorger, P. K. (2021) · Nature Methods · doi:10.1038/s41592-021-01283-4 · Cited by 96 · see citing works
Wang, J., Sun, T., Liu, B., Cao, Y., & Zhu, H. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/514 · Cited by 73 · see citing works
Teaching Machines to Ask Questions
Yao, K., Zhang, L., Luo, T., Tao, L., & Wu, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/632 · Cited by 44 · see citing works
Learning towards conversational AI: A survey
Fu, T., Gao, S., Zhao, X., Wen, J.-R., & Yan, R. (2022) · AI Open · doi:10.1016/j.aiopen.2022.02.001 · Cited by 46 · see citing works
Text-To-Text Generative Adversarial Networks
Li, C., Su, Y., & Liu, W. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489624 · Cited by 33 · see citing works
The Linear Representation Hypothesis and the Geometry of Large Language Models
Park, K., Choe, Y. J., & Veitch, V. (2023) · arXiv · doi:10.48550/arxiv.2311.03658 · Cited by 11 · see citing works
Multimodal Generative Models for Compositional Representation Learning
Wu, M., & Goodman, N. (2019) · arXiv · doi:10.48550/arxiv.1912.05075 · Cited by 8 · see citing works
- Authors
- Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., & Bengio, S.
- Affiliations
- Stanford University (US)
- University of Massachusetts Amherst (US)
- Alphabet (United States) (US)
- Google (United States) (US)
- Venue
- Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning
- Published
- 2016
- Pages
- 10-21
- OpenAlex
- W2210838531
- Cited by
- 2,061
- References
- 3
- Citations by year
- 2026: 68 · 2025: 95 · 2024: 136 · 2023: 165 · 2022: 243 · 2021: 364 · 2020: 358 · 2019: 341 · 2018: 187 · 2017: 83 · 2016: 21
- FWCI
- 113.69
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Computational Natural Language Learning (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- sentence generation, variational autoencoder, recurrent neural network language models, continuous latent space, deterministic decoding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
