Neural natural language generation: A survey on multilinguality, multimodality, controllability and learning
Open accessAbstract
Developing artificial learning systems that can understand and generate natural language has been one of the long-standing goals of artificial intelligence. Recent decades have witnessed an impressive progress on both of these problems, giving rise to a new family of approaches. Especially, the advances in deep learning over the past couple of years have led to neural approaches to natural language generation (NLG). These methods combine generative language learning techniques with neural-networks based frameworks. With a wide range of applications in natural language processing, neural NLG (NNLG) is a new and fast growing field of research. In this state-of-the-art report, we investigate the recent developments and applications of NNLG in its full extent from a multidimensional view, covering critical perspectives such as multimodality, multilinguality, controllability and learning strategies. We summarize the fundamental building blocks of NNLG approaches from these aspects and provide detailed reviews of commonly used preprocessing steps and basic neural architectures. This report also focuses on the seminal applications of these NNLG models such as machine translation, description generation, automatic speech recognition, abstractive summarization, text simplification, question answering and generation, and dialogue generation. Finally, we conclude with a thorough discussion of the described frameworks by pointing out some open research directions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In AI data security, key requirements and challenges include ensuring data confidentiality, adhering to consent norms, and safeguarding against vulnerabilities like membership inference attacks.
“In AI data security, key requirements and challenges include ensuring data confidentiality, adhering to consent norms, and safeguarding against vulnerabilities like membership inference attacks [228], [229].”
From From Google Gemini to OpenAI Q* (Q-Star): A Survey on Reshaping the Generative Artificial Intelligence (AI) Research Landscape (2025) · cites this paper as [228]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 390 · Cited by 54
Research works printed but not linked: 23
- Artetxe, M., Labaka, G., Agirre, E., & Cho, K. (2018). Unsupervised neural machine translation. In International Conference on Learning Representations.
- Bernsen, N. O. (2008). Multimodality theory. In Multimodal User Interfaces, pp. 5-29. Springer, Berlin, Heidelberg.
- Bhagat, R., & Hovy, E. (2013). What is a paraphrase?. Computational Linguistics, 39 (3), 463-472.
- Bi, B., Li, C., Wu, C., Yan, M., Wang, W., Huang, S., Huang, F., & Si, L. (2020). PALM: Pre-training an autoencoding&autoregressive language model for context-conditioned generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8681-8691.
- Chen, A., Stanovsky, G., Singh, S., & Gardner, M. (2020). MOCHA: A dataset for training and evaluating generative reading comprehension metrics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6521-6532.
- Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M., Parikh, D., & Batra, D. (2017). Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 326-335.
- Fisch, A., Lee, K., Chang, M. W., Clark, J. H., & Barzilay, R. (2020). CAPWAP: Captioning with a Purpose. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
- Graves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006). Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376.
- He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask r-cnn. In 2017 IEEE Inter- national Conference on Computer Vision (ICCV), pp. 2980-2988.
- Chaudhuri, K., & Salakhutdinov, R. (Eds.), Proceedings of the 36th International Con- ference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Vol. 97 of Proceedings of Machine Learning Research, pp. 2790-2799. PMLR.
- Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., & Xing, E. P. (2017). Toward controlled generation of text. In Precup, D., & Teh, Y. W. (Eds.), Proceedings of the 34th International Conference on Machine Learning, Vol. 70 of Proceedings of Machine Learning Research, pp. 1587-1596.
- Le, H., Sahoo, D., Chen, N., & Hoi, S. (2019). Multimodal transformer networks for end- to-end video-grounded dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5612-5623.
- Li, K., Li, J., Ye, G., Zhao, R., & Gong, Y. (2019). Towards code-switching asr for end-to- end ctc models. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6076-6080. IEEE.
- Libovický, J., Palaskar, S., Gella, S., & Metze, F. (2018). Multimodal abstractive sum- marization for open-domain videos. In Visually Grounded Interaction and Language (ViGIL), pp. 1-8.
- Liu, B., Wei, H., Niu, D., Chen, H., & He, Y. (2020). Asking questions the human way: Scalable question-answer generation from text corpus. In Proceedings of The Web Conference 2020, pp. 2032-2043.
- Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, L. (2016). Multi-Task Sequence to Sequence Learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2016, San Juan, Puerto Rico.
- Patil, C., & Patwardhan, M. (2020). Visual question generation: The state of the art. ACM Comput. Surv., 53 (3).
- Reed, S. E., Akata, Z., Mohan, S., Tenka, S., Schiele, B., & Lee, H. (2016). Learning what and where to draw. In Advances in neural information processing systems, pp. 217-225.
- Shu, L., Papangelis, A., Wang, Y.-C., Tur, G., Xu, H., Feizollahi, Z., Liu, B., & Molino, P. (2020). Controllable text generation with focused variation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3805-3817, Online. As- sociation for Computational Linguistics.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., & Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., & Garnett, R. (Eds.), Advances in Neural Information Processing Systems (NeurIPS), Vol. 30.
- Wang, Y., Zhang, J., Zhai, F., Xu, J., & Zong, C. (2018). Three strategies to improve one- to-many multilingual translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2955-2960. Association for Computa- tional Linguistics.
- Wang, Z., Wang, X., An, B., Yu, D., & Chen, C. (2020). Towards faithful neural table-to- text generation with content-matching constraints. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1072-1086.
- Wiriyathammabhum, P., Summers-Stay, D., Fermüller, C., & Aloimonos, Y. (2016). Com- puter vision and natural language processing: Recent approaches in multimedia and robotics. ACM Comput. Surv., 49 (4).
- Authors
- Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., Lloret, E., Apostol, E.-S., Truică, C.-O., Šandrih, B., Martinčić-Ipšić, S., Berend, G., Gatt, A., & Korvel, G.
- Affiliations
- Hacettepe University (TR)
- Heidelberg University (DE)
- IT University of Copenhagen (DK)
- Kharkiv National University of Radio Electronics (UA)
- Koç University (TR)
- New York University (US)
- University of Alicante (ES)
- Universitatea Națională de Știință și Tehnologie Politehnica București (RO)
- University of Belgrade (RS)
- University of Rijeka (HR)
- University of Szeged (HU)
- ORCID
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2022-04-06
- Volume
- 73
- Pages
- 1131-1207
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W4225593490
- Cited by
- 54
- References
- 390
- Citations by year
- 2026: 8 · 2025: 13 · 2024: 11 · 2023: 18 · 2022: 4
- FWCI
- 3.79
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Artificial Intelligence Research (publishedVersion, cc-by)
- Pure Amsterdam UMC (publishedVersion, other-oa)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- Repositorio Institucional de la Universidad de Alicante (Universidad de Alicante) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion)
- Funders
- European Commission (HORIZON2020, CA18231)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- multilingualism, multimodality, controllability, machine translation, description generation, speech recognition, abstractive summarization, text simplification, question answering, dialogue generation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
