Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation
Open accessAbstract
This paper surveys the current state of the art in Natural Language Generation (NLG), defined as the task of generating text or speech from non-linguistic input. A survey of NLG is timely in view of the changes that the field has undergone over the past two decades, especially in relation to new (usually data-driven) methods, as well as new applications of NLG technology. This survey therefore aims to (a) give an up-to-date synthesis of research on the core tasks in NLG and the architectures adopted in which such tasks are organised; (b) highlight a number of recent research topics that have arisen partly as a result of growing synergies between NLG and other areas of artificial intelligence; (c) draw attention to the challenges in NLG evaluation, relating them to similar challenges faced in other areas of NLP, with an emphasis on different evaluation methods and the relationships between them.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Lakoff, G. P., & Johnson, M. L. (2003) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226470993.001.0001 · Cited by 25,252 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221 · see citing works
Austin, J. L. (1975) · doi:10.1093/acprof:oso/9780198245537.001.0001 · Cited by 21,496 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 3× · see citing works
“Various standard lexical resources are used for joke production, including a British pronunciation dictionary (to find different words with a similar pronunciation, such as ‘bizarre’ and ‘bazaar’) and WordNet (Miller, 1995, to find words with a similar meaning, such as bazaar and market). jape uses various techniques…”
BackgroundInfluentialPoliteness : Some Universals in Language Usage
Brown, P. (1987) · Cited by 11,901 · see citing works
Searle, J. R. (1969) · doi:10.1017/cbo9781139173438 · Cited by 11,718 · cited 1× · see citing works
“This perspective on nlg is therefore related to the view of ‘language as action’ (Clark, 1996), itself rooted in a philosophical tradition inaugurated by the work of Austin (1962) and Searle (1969).”
BackgroundElman, J. L. (1990) · Cognitive Science · doi:10.1207/s15516709cog1402_1 · Cited by 11,212 · see citing works
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387 · see citing works
Matthiessen, C. M. I. M. (1989) · Language · doi:10.2307/414947 · Cited by 9,894 · see citing works
An Introduction to Functional Grammar
Halliday, M. A. K., Matthiessen, C. M. I. M., Halliday, M., & Matthiessen, C. (2014) · doi:10.4324/9780203783771 · Cited by 9,662 · see citing works
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · see citing works
(1999) · Cited by 7,995 · see citing works
Opinion Mining and Sentiment Analysis
Pang, B., & Lee, L. (2008) · now publishers, Inc. eBooks · doi:10.1561/9781601981516 · Cited by 6,957 · see citing works
Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
Oliva, A., & Torralba, A. B. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011139631724 · Cited by 6,321 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
The Narrative Construction of Reality
Bruner, J. (1991) · Critical Inquiry · doi:10.1086/448619 · Cited by 5,728 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Long-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · see citing works
Variation across Speech and Writing
Biber, D. (1988) · doi:10.1017/cbo9780511621024 · Cited by 5,227 · see citing works
A theory of lexical access in speech production [target paper]
Levelt, W. J. M., Roelofs, A., & Meyer, A. S. (1999) · Radboud Repository (Radboud University) · doi:10.1017/s0140525x99001776 · Cited by 5,173 · see citing works
Speaking: from intention to articulation
(1989) · Choice Reviews Online · doi:10.5860/choice.27-1947 · Cited by 5,016 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving
Fikes, R. E., & Nilsson, N. J. (1971) · Artificial Intelligence · doi:10.1016/0004-3702(71)90010-5 · Cited by 4,882 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
Caffe: Convolutional Architecture for Fast Feature EmbeddingPreprint
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 4,748 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
(2005) · Cited by 4,152 · see citing works
Speaking. From Intention to Articulation
Caplan, D. N. (1990) · Trends in Neurosciences · doi:10.1016/0166-2236(90)90151-y · Cited by 4,036 · see citing works
Speech Acts: An Essay in the Philosophy of Language.
Alston, W. P., & Searle, J. R. (1970) · The Philosophical Quarterly · doi:10.2307/2218090 · Cited by 3,911 · see citing works
Caltech-UCSD Birds-200-2011 Dataset
Wah, C. (2024) · CaltechAUTHORS (California Institute of Technology) · doi:10.57702/i9oeewf6 · Cited by 3,875 · see citing works
A Systematic Comparison of Various Statistical Alignment Models
Och, F. J., & Ney, H. (2003) · Computational Linguistics · doi:10.1162/089120103321337421 · Cited by 3,855 · see citing works
Rhetorical Structure Theory: Toward a functional theory of text organization
MANN, W. C., & THOMPSON, S. A. (1988) · Text - Interdisciplinary Journal for the Study of Discourse · doi:10.1515/text.1.1988.8.3.243 · Cited by 3,853 · see citing works
Automatic acquisition of hyponyms from large text corpora
Hearst, M. A. (1992) · International Conference on Computational Linguistics · doi:10.3115/992133.992154 · Cited by 3,302 · see citing works
Conducting behavioral research on Amazon’s Mechanical Turk
Mason, W., & Suri, S. (2012) · Behavior Research Methods · doi:10.3758/s13428-011-0124-6 · Cited by 3,172 · see citing works
Understanding natural language
Winograd, T. A. (1972) · Cognitive Psychology · doi:10.1016/0010-0285(72)90002-3 · Cited by 2,803 · see citing works
Toward a mechanistic psychology of dialogue.
Pickering, M. J., & Garrod, S. (2004) · Behavioral and Brain Sciences · doi:10.1017/s0140525x04000056 · Cited by 2,678 · see citing works
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,646 · see citing works
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,439 · see citing works
A Study of Translation Edit Rate with Targeted Human Annotation
Snover, M., Dorr, B. J., Schwartz, R. M., Micciulla, L., & Makhoul, J. I. (2006) · Cited by 2,380 · see citing works
Data collection in a flat world: the strengths and weaknesses of Mechanical Turk samples
Goodman, J. K., Cryder, C. E., & Cheema, A. (2013) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1753 · Cited by 2,242 · see citing works
Learning and development in neural networks: the importance of starting small
Elman, J. L. (1993) · Cognition · doi:10.1016/0010-0277(93)90058-4 · Cited by 1,988 · cited 1× · see citing works
“Since the early 1990s, when interest in neural approaches waned in the nlp and ai communities, cognitive science research has continued to explore their application to syntax and language production (e.g., Elman, 1990, 1993; Chang et al., 2006).”
BackgroundSPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
5.7 The WORDNET lexical database of English
Ooi, V. B. Y. (2019) · Edinburgh University Press eBooks · doi:10.1515/9781474471459-050 · Cited by 1,861 · see citing works
Centering: A Framework for Modelling the Local Coherence of Discourse,
Grosz, B. J., Joshi, A. K., & Weinstein, S. (1995) · doi:10.21236/ada324949 · Cited by 1,793 · see citing works
Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach
Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M. E. P., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0073791 · Cited by 1,700 · see citing works
From Word Embeddings To Document Distances
Kusner, M. J., Sun, Y., Kolkin, N., & Weinberger, K. Q. (2015) · PolyPublie (École Polytechnique de Montréal) · Cited by 1,695 · see citing works
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
Serban, I. V., Sordoni, A., Bengio, Y., Courville, A. C., & Pineau, J. (2016) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v30i1.9883 · Cited by 1,639 · see citing works
Object Recognition as Machine Translation: Learning a Lexicon for a Fixed Image Vocabulary
Duygulu, P., Barnard, K., Freitas, J., & Forsyth, D. A. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47979-1_7 · Cited by 1,624 · see citing works
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics
Doddington, G. R. (2002) · International Conference on Human Language Technology Research · doi:10.3115/1289189.1289273 · Cited by 1,603 · see citing works
Steedman, M. J. (2000) · The MIT Press eBooks · doi:10.7551/mitpress/6591.001.0001 · Cited by 1,601 · see citing works
Automatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. H. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073445.1073465 · Cited by 1,591 · see citing works
An integrated theory of language production and comprehension
Pickering, M. J., & Garrod, S. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12001495 · Cited by 1,569 · see citing works
Long-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547 · see citing works
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · IEEE International Conference on Computer Vision · doi:10.1109/iccv.2015.515 · Cited by 1,507 · see citing works
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
Hodosh, M., Young, P., & Hockenmaier, J. (2013) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3994 · Cited by 1,349 · see citing works
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,305 · see citing works
Recurrent Continuous Translation Models
Kalchbrenner, N., & Blunsom, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1176 · Cited by 1,277 · see citing works
Techniques for automatically correcting words in text
Kukich, K. (1992) · ACM Computing Surveys · doi:10.1145/146370.146380 · Cited by 1,270 · see citing works
A Maximum Entropy Model for Part-Of-Speech Tagging
Ratnaparkhi, A. (1996) · Cited by 1,268 · see citing works
Generating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · Cited by 1,266 · see citing works
Scale Structure, Degree Modification, and the Semantics of Gradable Predicates
Kennedy, C., & McNally, L. (2005) · Language · doi:10.1353/lan.2005.0071 · Cited by 1,262 · see citing works
A Primer on Neural Network Models for Natural Language Processing
Goldberg, Y. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4992 · Cited by 1,259 · see citing works
Every Picture Tells a Story: Generating Sentences from Images
Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15561-1_2 · Cited by 1,240 · see citing works
Building Natural-Language Generation SystemsPreprint
Reiter, E. (1996) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9605002 · Cited by 1,228 · see citing works
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.9 · Cited by 1,211 · see citing works
Heterogeneous Uncertainty Sampling for Supervised Learning
Lewis, D. D., & Catlett, J. (1994) · Elsevier eBooks · doi:10.1016/b978-1-55860-335-6.50026-x · Cited by 1,174 · see citing works
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.494 · Cited by 1,161 · see citing works
Narrative discourse : an essay in method
Genette, G., Lewin, J. E., & Culler, J. (1980) · Comparative Literature · doi:10.2307/1770890 · Cited by 1,087 · see citing works
Computer-based personality judgments are more accurate than those made by humans
Youyou, W., Kosinski, M., & Stillwell, D. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1418680112 · Cited by 1,086 · see citing works
ReferItGame: Referring to Objects in Photographs of Natural Scenes
Kazemzadeh, S., Ordóñez, V., Matten, M., & Berg, T. L. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1086 · Cited by 1,082 · see citing works
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048 · see citing works
Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments
Lavie, A., & Agarwal, A. (2007) · Proceedings of the Second Workshop on Statistical Machine Translation - StatMT '07 · doi:10.3115/1626355.1626389 · Cited by 1,047 · see citing works
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)Preprint
Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., & Yuille, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6632 · Cited by 1,039 · see citing works
A cognitive process theory of writing
Flower, L., & Hayes, J. R. (1981) · College Composition & Communication · doi:10.58680/ccc198115885 · Cited by 1,025 · see citing works
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R. J., & Saenko, K. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1173 · Cited by 996 · see citing works
Computational Interpretations of the Gricean Maxims in the Generation of Referring Expressions
Dale, R., & Reiter, E. (1995) · Cognitive Sciences · doi:10.1207/s15516709cog1902_3 · Cited by 931 · see citing works
A Persona-Based Neural Conversation Model
Li, J., Galley, M., Brockett, C., Spithourakis, G., Gao, J., & Dolan, B. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1094 · Cited by 898 · see citing works
Im2Text: Describing Images Using 1 Million Captioned Photographs
Ordóñez, V., Kulkarni, G. A., & Berg, T. L. (2011) · Cited by 879 · see citing works
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems
Wen, T.-H., Gašić, M., Mrkšić, N., Su, P.-H., Vandyke, D., & Young, S. J. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1199 · Cited by 876 · see citing works
Mani, I. (2001) · Natural language processing · doi:10.1075/nlp.3 · Cited by 841 · see citing works
BabyTalk: Understanding and Generating Simple Image Descriptions
Kulkarni, G. A., Premraj, V., Ordóñez, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.162 · Cited by 839 · see citing works
Deep Fragment Embeddings for Bidirectional Image Sentence MappingPreprint
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 802 · see citing works
Toward Controlled Generation of TextPreprint
Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., & Xing, E. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00955 · Cited by 802 · see citing works
Better Word Representations with Recursive Neural Networks for Morphology
Luong, M.-T., Socher, R., & Manning, C. D. (2013) · Conference on Computational Natural Language Learning · Cited by 789 · see citing works
An Introduction to Machine Translation
Hutchins, W. J., & SOMERS, H. L. (1992) · Medical Entomology and Zoology · Cited by 787 · see citing works
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 779 · see citing works
McKeown, K. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511620751 · Cited by 773 · see citing works
Amazon Mechanical Turk: Gold Mine or Coal Mine?
Fort, K., Adda, G., & Cohen, K. B. (2011) · Computational Linguistics · doi:10.1162/coli_a_00057 · Cited by 772 · see citing works
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J., Gao, J., & Dolan, B. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1020 · Cited by 768 · see citing works
Lin, C.-Y., & Och, F. J. (2004) · Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics - ACL '04 · doi:10.3115/1218955.1219032 · Cited by 737 · see citing works
Elements of a plan-based theory of speech acts
Cohen, P. R., & Perrault, C. R. (1979) · Cognitive Science · doi:10.1016/s0364-0213(79)80006-3 · Cited by 733 · see citing works
Magnitude Estimation of Linguistic Acceptability
Bard, E. G., Robertson, D., & Sorace, A. (1996) · Language · doi:10.2307/416793 · Cited by 724 · see citing works
Joshi, A. K., & Schabes, Y. (1997) · Springer eBooks · doi:10.1007/978-3-642-59126-6_2 · Cited by 722 · see citing works
A Survey of Techniques for Event Detection in Twitter
Farzindar, A., & Khreich, W. (2013) · Computational Intelligence · doi:10.1111/coin.12017 · Cited by 719 · see citing works
Search-Based Procedural Content Generation: A Taxonomy and Survey
Togelius, J., Yannakakis, G. N., Stanley, K. O., & Browne, C. (2011) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2011.2148116 · Cited by 706 · see citing works
Russell, S., Wefald, E. H., Bobrow, D. G., Brady, M., & Davis, R. (1991) · The MIT Press eBooks · doi:10.7551/mitpress/2474.001.0001 · Cited by 655 · see citing works
Building applied natural language generation systems
Reiter, E., & Dale, R. (1997) · Natural Language Engineering · doi:10.1017/s1351324997001502 · Cited by 653 · see citing works
Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images
Malinowski, M., Rohrbach, M., & Fritz, M. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.9 · Cited by 633 · see citing works
Re-evaluating the Role of Bleu in Machine Translation Research
Callison-Burch, C., Osborne, M., & Koehn, P. (2006) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 632 · see citing works
Understanding figurative language: from metaphors to idioms
(2002) · Choice Reviews Online · doi:10.5860/choice.39-3217 · Cited by 599 · see citing works
Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. (2014) · Cited by 593 · see citing works
Multi-Task Learning for Multiple Language Translation
Dong, D., Wu, H. R., He, W. P., Yu, D., & Wang, H. (2015) · doi:10.3115/v1/p15-1166 · Cited by 573 · see citing works
Three new graphical models for statistical language modelling
Mnih, A., & Hinton, G. E. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273577 · Cited by 569 · see citing works
Data-Driven Response Generation in Social Media
Ritter, A., Cherry, C. A., & Dolan, W. B. (2011) · Cited by 566 · see citing works
Linguistic Style Matching in Social Interaction
Niederhoffer, K. G., & Pennebaker, J. W. (2002) · Journal of Language and Social Psychology · doi:10.1177/026192702237953 · Cited by 563 · see citing works
Paraphrasing with bilingual parallel corpora
Bannard, C., & Callison-Burch, C. (2005) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1219840.1219914 · Cited by 535 · see citing works
Generating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_1 · Cited by 534 · see citing works
Evaluating Content Selection in Summarization: The Pyramid Method
Nenkova, A., & Passonneau, R. (2004) · Columbia University · doi:10.7916/d80r9xvd · Cited by 527 · see citing works
Narrative Planning: Balancing Plot and Character
Riedl, M. O., & Young, R. M. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2989 · Cited by 522 · see citing works
Baby talk: Understanding and generating simple image descriptions
Kulkarni, G. A., Premraj, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2011) · doi:10.1109/cvpr.2011.5995466 · Cited by 521 · see citing works
Dale, R. (2016) · Natural Language Engineering · doi:10.1017/s1351324916000243 · Cited by 488 · see citing works
Grounding Action Descriptions in Videos
Regneri, M., Rohrbach, M., Wetzel, D., Thater, S., Schiele, B., & Pinkal, M. (2013) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00207 · Cited by 471 · see citing works
Nenkova, A. (2011) · now publishers, Inc. eBooks · doi:10.1561/9781601984715 · Cited by 457 · see citing works
Controlling Linguistic Style Aspects in Neural Language Generation
Ficler, J., & Goldberg, Y. (2017) · arXiv.org · doi:10.18653/v1/w17-4912 · Cited by 432 · see citing works
Midge: Generating Image Descriptions From Computer Vision Detections
Mitchell, M., Dodge, J., Goyal, A., Yamaguchi, K., Stratos, K., Han, X., Mensch, A., Berg, A., Berg, T. L., & Daumé, H. (2012) · Cited by 421 · see citing works
CCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank
Hockenmaier, J., & Steedman, M. J. (2007) · Computational Linguistics · doi:10.1162/coli.2007.33.3.355 · Cited by 389 · see citing works
A Survey of Paraphrasing and Textual Entailment Methods
Androutsopoulos, I., & Malakasiotis, P. (2010) · Cited by 386 · see citing works
Using Speakers' Referential Intentions to Model Early Cross-Situational Word Learning
Frank, M. C., Goodman, N. D., & Tenenbaum, J. B. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02335.x · Cited by 385 · see citing works
Computational Generation of Referring Expressions: A Survey
Krahmer, E., & van Deemter, K. (2012) · International Conference on Computational Logic · doi:10.1162/coli_a_00088 · Cited by 384 · see citing works
Self-citation(Meta-) evaluation of machine translation
Callison-Burch, C., Fordyce, C. S., Koehn, P., Monz, C., & Schroeder, J. (2007) · Conference on Machine Translation (WMT) · doi:10.3115/1626355.1626373 · Cited by 381 · see citing works
Enter the Robot Journalist: Users' perceptions of automated content
Clerwall, C. (2014) · Journalism Practice · doi:10.1080/17512786.2014.883116 · Cited by 377 · see citing works
Corpus-Guided Sentence Generation of Natural Images
Yang, Y., Teo, C. L., Daumé, H., & Aloimonos, Y. (2011) · Cited by 367 · see citing works
Composing Simple Image Descriptions using Web-scale N-grams
Li, S., Kulkarni, G. A., Berg, T. L., Berg, A. C., & Choi, Y. (2011) · Cited by 364 · see citing works
Kojima, A., Tamura, T., & Fukunaga, K. (2002) · International Journal of Computer Vision · doi:10.1023/a:1020346032608 · Cited by 359 · see citing works
Appelt, D. E. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511624575 · Cited by 357 · see citing works
Catching the Drift: Probabilistic Content Models, with Applications to Generation and SummarizationPreprint
Barzilay, R., & Lee, L. (2004) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0405039 · Cited by 355 · see citing works
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Raffaella Bernardi, B., Çakıcı, R., Elliott, D., Erdem, A., Erdem, E., İkizler-Cinbiş, N., Keller, F., Muscat, A., & Plank, B. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4900 · Cited by 350 · see citing works
What to do about bad language on the internet
Eisenstein, J. (2013) · Cited by 343 · see citing works
Generating natural language under pragmatic constraints
Hovy, E. H. (1987) · Journal of Pragmatics · doi:10.1016/0378-2166(87)90109-3 · Cited by 340 · see citing works
Planning text for advisory dialogues: capturing intentional and rhetorical information
Moore, J. D., & Paris, C. L. (1993) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 335 · see citing works
Inferring Strategies for Sentence Ordering in Multidocument News Summarization
Barzilay, R., & Elhadad, N. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.991 · Cited by 333 · see citing works
Do speakers and listeners observe the Gricean Maxim of Quantity?
Engelhardt, P. E., BAILEY, K., & FERREIRA, F. (2006) · Journal of Memory and Language · doi:10.1016/j.jml.2005.12.009 · Cited by 333 · see citing works
THE ALGORITHMS BEHIND THE HEADLINES
van Dalen, A. (2012) · Journalism Practice · doi:10.1080/17512786.2012.667268 · Cited by 330 · see citing works
Producing spoken language: a blueprint of the speaker
Levelt, W. J. M. (2000) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198507932.003.0004 · Cited by 323 · see citing works
Visual Turing test for computer vision systems
Geman, D., Geman, S., Hallonquist, N., & Laurent Younes, B. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1422953112 · Cited by 323 · see citing works
Collective Generation of Natural Image Descriptions
Кузнецова, П., Ordóñez, V., Berg, A., Berg, T. L., & Choi, Y. (2012) · Cited by 320 · see citing works
Automatic generation of textual summaries from neonatal intensive care data
Portet, F., Reiter, E., Albert Gatt, A., Jim Hunter, A., Sripada, S. G., Freer, Y., & Cindy Sykes, B. (2008) · Artificial Intelligence · doi:10.1016/j.artint.2008.12.002 · Cited by 318 · see citing works
TALE-SPIN, an interactive program that writes stories
Meehan, J. (1977) · Cited by 315 · see citing works
Learning semantic correspondences with less supervision
Liang, P., Jordan, M. I., & Klein, D. (2009) · doi:10.3115/1687878.1687893 · Cited by 314 · see citing works
I2T: Image Parsing to Text Description
Yao, B., Yang, X., Lin, L., Lee, M. W., & Zhu, S. C. (2010) · Proceedings of the IEEE · doi:10.1109/jproc.2010.2050411 · Cited by 310 · see citing works
Building Natural Language Generation Systems
Horacek, H. (2001) · Artificial Intelligence in Medicine · doi:10.1016/s0933-3657(00)00114-7 · Cited by 310 · see citing works
Automated discourse generation using discourse structure relations
Hovy, E. H. (1993) · Artificial Intelligence · doi:10.1016/0004-3702(93)90021-3 · Cited by 307 · see citing works
Chinese Poetry Generation with Recurrent Neural Networks
Zhang, X., & Lapata, M. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1074 · Cited by 298 · see citing works
The 1998 AI Planning Systems Competition
McDermott, D. M. (2000) · The AI Magazine · doi:10.1609/aimag.v21i2.1506 · Cited by 294 · see citing works
Image Description using Visual Dependency Representations
Elliott, D., & Keller, F. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1128 · Cited by 282 · see citing works
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data
Hendricks, L. A., Venugopalan, S., Rohrbach, M., Mooney, R. J., Saenko, K., & Darrell, T. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.8 · Cited by 280 · see citing works
Making preferences more active
Wilks, Y. A. (1978) · Artificial Intelligence · doi:10.1016/0004-3702(78)90001-2 · Cited by 277 · see citing works
(1996) · Medical Entomology and Zoology · Cited by 276 · see citing works
Generating Natural Questions About an Image
Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., & Vanderwende, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1170 · Cited by 270 · see citing works
Further meta-evaluation of machine translation
Callison-Burch, C., Fordyce, C. S., Koehn, P., Monz, C., & Schroeder, J. (2008) · Conference on Machine Translation (WMT) · doi:10.3115/1626394.1626403 · Cited by 268 · see citing works
T ree T alk : Composition and Compression of Trees for Image Descriptions
Kuznetsova, P., Ordóñez, V., Berg, T. L., & Choi, Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00188 · Cited by 268 · see citing works
Using natural-language processing to produce weather forecasts
Goldberg, E., Driedger, N., & Kittredge, R. I. (1994) · IEEE Expert · doi:10.1109/64.294135 · Cited by 265 · see citing works
Near-Synonymy and Lexical Choice
Edmonds, P., & Hirst, G. (2002) · Computational Linguistics · doi:10.1162/089120102760173625 · Cited by 262 · see citing works
Choosing words in computer-generated weather forecasts
Reiter, E., Sripada, S. G., Jim Hunter, A., Yu, J. S., & Davy, I. P. (2005) · Artificial Intelligence · doi:10.1016/j.artint.2005.06.006 · Cited by 262 · see citing works
What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment
Mei, H., Bansal, M., & Walter, M. R. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1086 · Cited by 251 · see citing works
Evaluation of Machine Translation and its Evaluation
Turian, J. P., Shea, L., & Melamed, I. D. (2006) · doi:10.21236/ada453509 · Cited by 250 · see citing works
Automatic question generation for vocabulary assessment
Brown, J. C., Frishkoff, G. A., & Eskénazi, M. (2005) · doi:10.3115/1220575.1220678 · Cited by 250 · see citing works
Computational Linguistics and Deep Learning
Manning, C. D. (2015) · Computational Linguistics · doi:10.1162/coli_a_00239 · Cited by 246 · see citing works
Scripts, Sequences, and Stories: Elements of a Postclassical Narratology
Herman, D. (1997) · PMLA/Publications of the Modern Language Association of America · doi:10.2307/463482 · Cited by 244 · see citing works
Sentence Simplification by Monolingual Machine Translation
Wubben, S., van den Bosch, A. P. J., & Krahmer, E. (2012) · Research portal (Tilburg University) · Cited by 237 · see citing works
Self-citationMaking Preferences More Active
Wilks, Y. A. (2007) · Springer eBooks · doi:10.1007/1-4020-5285-5_7 · Cited by 236 · see citing works
Centering: A Parametric Theory and Its Instantiations
Poesio, M., Stevenson, R. J., Eugenio, B. D., & Hitzeman, J. M. (2004) · Computational Linguistics · doi:10.1162/0891201041850911 · Cited by 235 · see citing works
An architecture for data-to-text systems
Reiter, E. (2007) · European Workshop on Natural Language Generation (ENLG) · doi:10.3115/1610163.1610180 · Cited by 229 · see citing works
Lessons from a failure: Generating tailored smoking cessation letters
Reiter, E., Robertson, R., & Osman, L. M. (2003) · Artificial Intelligence · doi:10.1016/s0004-3702(02)00370-3 · Cited by 229 · see citing works
Paraphrasing for automatic evaluation
Kauchak, D., & Barzilay, R. (2006) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1220835.1220893 · Cited by 222 · see citing works
Cooking up referring expressions
Dale, R. (1989) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/981623.981632 · Cited by 219 · see citing works
A Computational Model of Metaphor Interpretation
Martin, J. (1990) · Cited by 214 · see citing works
Oberlander, J., & Nowson, S. (2006) · International Conference on Computational Linguistics and Annual Meeting of the Association for Computational Linguistics (COLING/ACL) · doi:10.3115/1273073.1273154 · Cited by 211 · see citing works
Mining the Blogosphere: Age, gender and the varieties of self-expression
Argamon, S., Koppel, M., Pennebaker, J. W., & Schler, J. (2007) · First Monday · doi:10.5210/fm.v12i9.2003 · Cited by 210 · cited 3× · see citing works
“…phenomenon, the linguistic features of personality are only indirectly reflected in speaking or writing (a hypothesis underlying much work on detection of personality and other features in text, including Oberlander & Nowson, 2006; Argamon et al., 2007; Schwartz et al., 2013; Youyou et al., 2015).”
BackgroundMethodChen, D. L., & Mooney, R. J. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390173 · Cited by 210 · see citing works
met*: a method for discriminating metonymy and metaphor by computer
Fass, D. (1991) · Cited by 207 · see citing works
Language Models for Image Captioning: The Quirks and What Works
Devlin, J., Cheng, H., Fang, H., Gupta, S., Deng, L., He, X., Zweig, G., & Mitchell, M. (2015) · doi:10.3115/v1/p15-2017 · Cited by 207 · see citing works
Real versus Template-Based Natural Language Generation: A False Opposition?
van Deemter, K., Theune, M., & Krahmer, E. (2005) · Computational Linguistics · doi:10.1162/0891201053630291 · Cited by 207 · see citing works
Self-citationReiter, E., & Belz, A. (2009) · Computational Linguistics · doi:10.1162/coli.2009.35.4.35405 · Cited by 204 · see citing works
Affective Neural Response Generation
Asghar, N., Poupart, P., Hoey, J., Jiang, X., & Mou, L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-76941-7_12 · Cited by 203 · see citing works
A survey of research on text simplification
Siddharthan, A. (2014) · ITL Review of Applied Linguistics · doi:10.1075/itl.165.2.06sid · Cited by 202 · see citing works
A fast and portable realizer for text generation systems
Lavoie, B., & Rambow, O. (1997) · Conference on Applied Natural Language Processing · doi:10.3115/974557.974596 · Cited by 202 · see citing works
Minstrel: a computer model of creativity and storytelling
Turner, R. S. (1993) · Cited by 198 · see citing works
Bringing Semantics into Focus Using Visual Abstraction
Zitnick, C. L., & Parikh, D. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.387 · Cited by 197 · see citing works
Automatic Evaluation of Information Ordering: Kendall's Tau
Lapata, M. (2006) · Computational Linguistics · doi:10.1162/coli.2006.32.4.471 · Cited by 194 · see citing works
Adaptive, intelligent presentation of information for the museum visitor in PEACH
Stock, O., Zancanaro, M., Busetta, P., Callaway, C. B., Krüger, A., Kruppa, M., Kuflik, T., Not, E., & Rocchi, C. (2007) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-007-9029-6 · Cited by 193 · see citing works
Learning visually grounded words and syntax for a scene description task
Roy, D. (2002) · Computer Speech & Language · doi:10.1016/s0885-2308(02)00024-4 · Cited by 193 · see citing works
Collective content selection for concept-to-text generation
Barzilay, R., & Lapata, M. (2005) · doi:10.3115/1220575.1220617 · Cited by 192 · see citing works
Generating and evaluating evaluative arguments
Carenini, G., & Moore, J. D. (2006) · Artificial Intelligence · doi:10.1016/j.artint.2006.05.003 · Cited by 191 · see citing works
Enabling technology for multilingual natural language generation: the KPML development environment
Bateman, J. A. (1997) · Natural Language Engineering · doi:10.1017/s1351324997001514 · Cited by 191 · see citing works
Learning to Generate Product Reviews from Attributes
Dong, L., Huang, S., Wei, F., Lapata, M., Zhou, M., & Xu, K. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1059 · Cited by 190 · see citing works
Artificial Intelligence and Literary Creativity: Inside the Mind of Brutus, A Storytelling Machine
Bringsjord, S., & Ferrucci, D. A. (1999) · Cited by 187 · see citing works
Not exactly: in praise of vagueness
(2010) · Choice Reviews Online · doi:10.5860/choice.48-0189 · Cited by 186 · see citing works
Callaway, C. B., & Lester, J. C. (2002) · Artificial Intelligence · doi:10.1016/s0004-3702(02)00230-8 · Cited by 184 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Stepin, I., Alonso, J. M., Catala, A., & Pereira-Farina, M. (2021) · IEEE Access · doi:10.1109/access.2021.3051315 · Cited by 376 · see citing works
A Survey of Knowledge-enhanced Text Generation
Yu, W., Zhu, C., Li, Z., Hu, Z., Wang, Q., Ji, H., & Jiang, M. (2022) · ACM Computing Surveys · doi:10.1145/3512467 · Cited by 257 · see citing works
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · see citing works
“Gkatzia and Mahamood (2015) focused on NLG papers from 2005-2014; Amidei et al. (2018a) provided a 2013-2018 overview of evaluation in question generation; and Gatt and Krahmer (2018) provided a more general survey of the state-of-the-art in NLG.”
BackgroundInfluentialSelf-citationA Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235 · see citing works
Conversational Agents: Goals, Technologies, Vision and Challenges
Allouch, M., Azaria, A., & Azoulay, R. (2021) · Sensors · doi:10.3390/s21248448 · Cited by 168 · see citing works
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159 · see citing works
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Dhingra, B., Faruqui, M., Parikh, A., Chang, M.-W., Das, D., & Cohen, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1483 · Cited by 150 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70 · see citing works
e-ViL: A Dataset and Benchmark for Natural Language Explanations in Vision-Language Tasks
Kayser, M., Camburu, O.-M., Salewski, L., Emde, C., Do, V., Akata, Z., & Lukasiewicz, T. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00128 · Cited by 68 · see citing works
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54 · see citing works
Self-citationData-to-text generation with macro planning
Puduppully, R., & Lapata, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00381 · Cited by 53 · see citing works
Controlling hallucinations at word level in data-to-text generation
Rebuffel, C., Roberti, M., Soulier, L., Scoutheeten, G., Cancelliere, R., & Gallinari, P. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-021-00801-4 · Cited by 50 · see citing works
Woo, D. J., Wang, Y., Susanto, H., & Guo, K. (2023) · Journal of Educational Computing Research · doi:10.1177/07356331231175999 · Cited by 44 · see citing works
Gubelmann, R. (2024) · Philosophy & Technology · doi:10.1007/s13347-024-00696-1 · Cited by 36 · see citing works
Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation
Rebuffel, C., Scialom, T., Soulier, L., Piwowarski, B., Lamprier, S., Staiano, J., Scoutheeten, G., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.633 · Cited by 24 · see citing works
The statistical advantage of automatic NLG metrics at the system level
Wei, J., & Jia, R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.533 · Cited by 16 · see citing works
“We do not give suggestions on any particular goals, and practitioners should understand what their application is, and which evaluation is the best approximation (refer to Gatt and Krahmer, 2018).”
BackgroundThe Pitfalls of Defining Hallucination
Deemter, K. V. (2024) · Computational Linguistics · doi:10.1162/coli_a_00509 · Cited by 13 · see citing works
Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
Fan, X., Lyu, Y., Liang, P. P., Salakhutdinov, R., & Morency, L.-P. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.758 · Cited by 3 · see citing works
- Authors
- Gatt, A., & Krahmer, E.
- Affiliations
- Tilburg University (NL)
- ORCID
- Krahmer, Emiel
- Venue
- Journal of Artificial Intelligence Research
- Publisher
- AI Access Foundation
- Published
- 2018-01-27
- Volume
- 61
- Pages
- 65-170
- ISSN
- 1076-9757 (ISSN-L)
- OpenAlex
- W2604799547
- Cited by
- 776
- References
- 554
- Citations by year
- 2026: 43 · 2025: 56 · 2024: 60 · 2023: 93 · 2022: 102 · 2021: 135 · 2020: 116 · 2019: 113 · 2018: 48 · 2017: 10
- FWCI
- 57.66
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Artificial Intelligence Research (publishedVersion, cc-by)
- Research portal (Tilburg University) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- OAR@UM (University of Malta) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (360-89-050)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- natural language generation, evaluation metrics, speech generation, interdisciplinary synergy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
