A Survey of Natural Language Generation
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
This article offers a comprehensive review of the research on Natural Language Generation (NLG) over the past two decades, especially in relation to data-to-text generation and text-to-text generation deep learning methods, as well as new applications of NLG technology. This survey aims to (a) give the latest synthesis of deep learning research on the NLG core tasks, as well as the architectures adopted in the field; (b) detail meticulously and comprehensively various NLG tasks and datasets, and draw attention to the challenges in NLG evaluation, focusing on different evaluation methods and their relationships; (c) highlight some future emphasis and relatively recent research issues that arise due to the increasing synergy between NLG and other artificial intelligence areas, such as computer vision, text, and computational creativity.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582 · cited 3×
“The graph neural network (GNN) [109] is used to process the data in graph form,which can capture the dependency information between nodes of a graph via message passing.”
MethodInfluentialLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 4×
“The text abbreviation task is formulated to condense information from long texts to short ones, typically including research on text summarization [8, 9, 20, 24, 53, 96, 116], question generation [5, 26, 44, 46, 66, 112, 122, 133, 134, 153, 158], and distractor generation [30, 61, 76, 89, 98, 103, 117, 118].”
BackgroundInfluentialMining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 5×
“Image captioning [110] has achieved significant improvements based on neural encoderdecoder framework.”
BackgroundMethodInfluentialSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“The Stanford Question Answering Dataset (SQuAD) [100] is a large reading comprehension dataset created by crowdworkers.”
BackgroundLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · cited 3×
“RACE.”
BackgroundInfluentialBLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.704 · Cited by 1,005
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005 · cited 1×
“The Bilingual Evaluation Understudy with Representations from Transformers (BLEURT) metric [111] is based on BERT [18] with a novel pre-training scheme.”
MethodGatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776 · cited 1×
“[31] introduce the core tasks, applications and evaluation metrics of natural language generation.”
BackgroundImproved Image Captioning via Policy Gradient optimization of SPIDEr
Liu, S., Zhu, Z., Ye, N., Guadarrama, S., & Murphy, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.100 · Cited by 430 · cited 1×
“To reduce exposure bias and metric mismatching in sequential training, notable eforts are made to optimise non-diferentiable metrics using reinforcement learning [68, 104, 144].”
MethodTexygen: A Benchmarking Platform for Text Generation Models
Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., & Yu, Y. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210080 · Cited by 408 · cited 1×
“The Self-BLEU metric [162] is also a metric to measure the diversity.”
BackgroundStyle transfer through back-translation
Prabhumoye, S., Tsvetkov, Y., Salakhutdinov, R., & Black, A. W. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1080 · Cited by 348 · cited 3×
“There are two sub-tasks: text style transfer [13, 28, 43, 63, 72, 80, 85, 95, 143, 157] and dialogue generation [4, 45, 57, 60, 74, 136, 145, 159].”
BackgroundMethodTowards an Automatic Turing Test: Learning to Evaluate Dialogue Responses
Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., & Pineau, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1103 · Cited by 341 · cited 1×
“The Automatic Dialogue Evaluation Model (ADEM) metric [69] is used to automatically evaluate the quality of dialogue responses, where the evaluation model is trained in a semi-supervised manner with a hierarchical recurrent neural network (RNN) to predict the response scores.”
MethodGenerating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus
Serban, I. V., García-Durán, A., Gulcehre, C., Ahn, S., Chandar, S., Courville, A., & Bengio, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1056 · Cited by 280 · cited 3×
“Question Generation Key-Phrase-based Question Generator [26] Keyphrase + Template Dynamic Mathematical Question Generator [5] Constraint Handling Rules KB-based Factoid Question Generator [112] RNN Teacher Forcing and RL Based Question Generator [153] RNN + RL Paragraph-level Question Generator [158] RNN Answer-Positi…”
BackgroundMethodEvaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · arXiv · doi:10.48550/arxiv.1801.07243 · Cited by 165
Shakespearizing Modern Language Using Copy-Enriched Sequence to Sequence Models
Jhamtani, H., Gangal, V., Hovy, E., & Nyberg, E. (2017) · Proceedings of the Workshop on Stylistic Variation · doi:10.18653/v1/w17-4902 · Cited by 156 · cited 3×
“There are two sub-tasks: text style transfer [13, 28, 43, 63, 72, 80, 85, 95, 143, 157] and dialogue generation [4, 45, 57, 60, 74, 136, 145, 159].”
BackgroundMethodFactual Error Correction for Abstractive Summarization Models
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.506 · Cited by 132
Cited by
Machine-Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E. N., Japkowicz, N., & Viktor, H. L. (2023) · IEEE Access · doi:10.1109/access.2023.3294090 · Cited by 196
A Review on Edge Large Language Models: Design, Execution, and Applications
Zheng, Y., Chen, Y., Qian, B., Shi, X., Shu, Y., & Chen, J. (2025) · ACM Computing Surveys · doi:10.1145/3719664 · Cited by 169
Explaining machine learning models with interactive natural language conversations using TalkToModel
Slack, D., Krishna, S., Lakkaraju, H., & Singh, S. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00692-8 · Cited by 125
Can large language models write reflectively
Li, Y., Sha, L., Yan, L., Lin, J., Raković, M., Galbraith, K., Lyons, K., Gašević, D., & Chen, G. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100140 · Cited by 99
Self-citationMapping the Design Space of Human-AI Interaction in Text Summarization
Cheng, R., Smith-Renner, A., Zhang, K., Tetreault, J., & Jaimes-Larrarte, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.33 · Cited by 20
Metadata
- Authors
- Dong, C., Li, Y., Gong, H., Chen, M., Li, J., Shen, Y., & Yang, M.
- Venue
- ACM Computing Surveys
- Published
- 2022-12-23
- Volume
- 55
- Issue
- 8
- Pages
- 1-38
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4226031686
- Cited by
- 149
- References
- 30
- FWCI
- 10.42
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.