Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Open accessAbstract
Automatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio (Lebret et al., 2016), often contain reference texts that diverge from the information in the corresponding semistructured data.We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge.We propose a new metric, PAR-ENT, which aligns n-grams from the reference and generated texts to the semi-structured data before computing their precision and recall.Through a large scale human evaluation study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics.We also adapt and evaluate the information extraction based evaluation proposed in Wiseman et al. (2017), and show that PAR-ENT has comparable correlation to it, while being easier to use.We show that PARENT is also applicable when the reference texts are elicited from humans using the data from the WebNLG challenge.1 * Work done during an internship at Google.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
(2004) · Cited by 9,718 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · see citing works
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · see citing works
Automatic evaluation of machine translation quality using n-gram co-occurrence statistics
Doddington, G. R. (2002) · International Conference on Human Language Technology Research · doi:10.3115/1289189.1289273 · Cited by 1,603 · see citing works
Liu, C., Lowe, R., Serban, I. V., Noseworthy, M., Charlin, L., & Pineau, J. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1230 · Cited by 1,262 · see citing works
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776 · see citing works
McKeown, K. R. (1985) · Cambridge University Press eBooks · doi:10.1017/cbo9780511620751 · Cited by 773 · see citing works
Building applied natural language generation systems
Reiter, E., & Dale, R. (1997) · Natural Language Engineering · doi:10.1017/s1351324997001502 · Cited by 653 · see citing works
Re-evaluating the Role of Bleu in Machine Translation Research
Callison-Burch, C., Osborne, M., & Koehn, P. (2006) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 632 · see citing works
Neural Text Generation from Structured Data with Application to the Biography Domain
Lebret, R., Grangier, D., & Auli, M. (2016) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1128 · Cited by 595 · see citing works
The pascal recognising textual entailment challenge
Dagan, I., & Glickman, O. (2005) · Cited by 579 · see citing works
Optimizing Statistical Machine Translation for Text Simplification
Xu, W. H., Napoles, C., Pavlick, E., Chen, Q. Z., & Callison-Burch, C. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00107 · Cited by 524 · see citing works
Challenges in Data-to-Document Generation
Wiseman, S., Shieber, S., & Rush, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1239 · Cited by 524 · see citing works
Why We Need New Evaluation Metrics for NLG
Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1238 · Cited by 422 · see citing works
Object Hallucination in Image Captioning
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., & Saenko, K. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1437 · Cited by 420 · see citing works
A Structured Review of the Validity of BLEU
Reiter, E. (2018) · Computational Linguistics · doi:10.1162/coli_a_00322 · Cited by 386 · see citing works
(Meta-) evaluation of machine translation
Callison-Burch, C., Fordyce, C. S., Koehn, P., Monz, C., & Schroeder, J. (2007) · Conference on Machine Translation (WMT) · doi:10.3115/1626355.1626373 · Cited by 381 · see citing works
Learning semantic correspondences with less supervision
Liang, P., Jordan, M. I., & Klein, D. (2009) · doi:10.3115/1687878.1687893 · Cited by 314 · see citing works
Design of a knowledge-based report generator
Kukich, K. (1983) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/981311.981340 · Cited by 211 · see citing works
Evaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge
Dušek, O., Novikova, J., & Rieser, V. (2019) · Computer Speech & Language · doi:10.1016/j.csl.2019.06.009 · Cited by 194 · see citing works
Collective content selection for concept-to-text generation
Barzilay, R., & Lapata, M. (2005) · doi:10.3115/1220575.1220617 · Cited by 192 · see citing works
Comparing automatic and human evaluation of NLG systems
Belz, A., & Reiter, E. (2006) · University of Brighton Repository (University of Brighton) · Cited by 182 · see citing works
Re-evaluating Automatic Metrics for Image Captioning
Kilickaya, M., Erdem, A., İkizler-Cinbiş, N., & Erdem, E. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1019 · Cited by 149 · see citing works
Evaluation metrics for generation
Bangalore, S., Rambow, O., & Whittaker, S. (2000) · doi:10.3115/1118253.1118255 · Cited by 127 · see citing works
Evaluating Evaluation Methods for Generation in the Presence of Variation
Stent, A., Marge, M., & Singhai, M. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30586-6_38 · Cited by 126 · see citing works
How to Analyze Paired Comparison Data
Tsukida, K., & Gupta, M. R. (2011) · Cited by 122 · see citing works
(2018) · Cited by 115 · see citing works
Towards a Better Metric for Evaluating Question Generation Systems
Nema, P., & Khapra, M. M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1429 · Cited by 106 · see citing works
Comparing dependent robust correlations
Wilcox, R. R. (2016) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/bmsp.12069 · Cited by 101 · see citing works
The price of debiasing automatic metrics in natural language evalaution
Chaganty, A., Mussmann, S., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1060 · Cited by 96 · see citing works
Joint Learning of a Dual SMT System for Paraphrase Generation
Sun, H. M., & Zhou, M. (2012) · Cited by 72 · see citing works
Testing for Significance of Increased Correlation with Human Judgment
Graham, Y., & Baldwin, T. J. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1020 · Cited by 67 · see citing works
Detecting Cross-Lingual Semantic Divergence for Neural Machine Translation
Carpuat, M. J., Vyas, Y., & Niu, X. (2017) · doi:10.18653/v1/w17-3209 · Cited by 59 · see citing works
Identifying Semantic Divergences in Parallel Text without Annotations
Vyas, Y., Niu, X., & Carpuat, M. J. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1136 · Cited by 36 · see citing works
PEM: A Paraphrase Evaluation Metric Exploiting Parallel Texts
Liu, C., Dahlmeier, D., & Ng, H. T. (2010) · Cited by 28 · see citing works
Improving Slot Filling Performance with Attentive Neural Networks on Dependency Structures
Huang, L., Sil, A., Ji, H., & Florian, R. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1274 · Cited by 28 · see citing works
A probabilistic setting and lexical cooccurrence model for textual entailment
Glickman, O., & Dagan, I. (2005) · doi:10.3115/1631862.1631870 · Cited by 21 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence- level bleu. In Proc. of WMT.
- Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating train- ing corpora for micro-planners. In Proc. of ACL.
- Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017b. The E2E dataset: New challenges for end- to-end generation. In Proc. of SIGDIAL.
- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Å Ąukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144. · arXiv
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
“PARENT [22] measures the accuracy of table-to-text generation by aligning n-grams from the reference description R and generated texts G to the table T .”
BackgroundMethodInfluentialOn Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185 · see citing works
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563 · see citing works
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.397 · Cited by 343 · see citing works
ToTTo: A Controlled Table-To-Text Generation Dataset
Parikh, A., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.89 · Cited by 252 · see citing works
Self-citationIs chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235 · see citing works
A Survey of Evaluation Metrics Used for NLG Systems
Sai, A. B., Mohankumar, A. K., & Khapra, M. M. (2023) · ACM Computing Surveys · doi:10.1145/3485766 · Cited by 235 · see citing works
Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports
Zhang, Y., Merck, D., Tsai, E., Manning, C. D., & Langlotz, C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.458 · Cited by 171 · see citing works
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159 · see citing works
Self-citationText-to-Text Pre-Training for Data-to-Text Tasks
Kale, M., & Rastogi, A. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.14 · Cited by 145 · see citing works
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
Few-Shot NLG with Pre-Trained Language Model
Chen, Z., Eavani, H., Chen, W., Liu, Y., & Wang, W. Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.18 · Cited by 127 · see citing works
Logical Natural Language Generation from Open-Domain Tables
Chen, W., Chen, J., Su, Y., Chen, Z., & Wang, W. Y. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.708 · Cited by 130 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
Planning with Learned Entity Prompts for Abstractive Summarization
Narayan, S., Zhao, Y., Maynez, J., Simões, G., Nikolaev, V., & McDonald, R. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00438 · Cited by 112 · see citing works
DART: Open-Domain Structured Data Record to Text Generation
Nan, L., Radev, D., Zhang, R., Rau, A., Sivaprasad, A., Hsieh, C., Tang, X., Vyas, A., Verma, N., Krishna, P., et al. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.37 · Cited by 89 · see citing works
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
Dziri, N., Madotto, A., Zaïane, O., & Bose, A. J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.168 · Cited by 81 · see citing works
INFOTABS: Inference on Tables as Semi-structured Data
Gupta, V., Mehta, M., Nokhiz, P., & Srikumar, V. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.210 · Cited by 81 · see citing works
Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data
Filippova, K. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.76 · Cited by 79 · see citing works
Improved Natural Language Generation via Loss Truncation
Kang, D. D., & Hashimoto, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.66 · Cited by 71 · see citing works
Plan-then-Generate: Controlled Data-to-Text Generation via Planning
Su, Y., Vandyke, D., Wang, S., Fang, Y., & Collier, N. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.76 · Cited by 67 · see citing works
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., & Dolan, B. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.464 · Cited by 63 · see citing works
HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation
Cheng, Z., Dong, H., Wang, Z., Jia, R., Guo, J., Gao, Y., Han, S., Lou, J., & Zhang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.78 · Cited by 63 · see citing works
Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation
Tian, R., Narayan, S., Sellam, T., & Parikh, A. P. (2019) · arXiv · doi:10.48550/arxiv.1910.08684 · Cited by 60 · see citing works
Self-citationTowards Table-to-Text Generation with Numerical Reasoning
Suadaa, L. H., Kamigaito, H., Funakoshi, K., Okumura, M., & Takamura, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.115 · Cited by 52 · see citing works
(2020) · doi:10.18653/v1/2020.findings-emnlp.190 · Cited by 51 · see citing works
Controlling hallucinations at word level in data-to-text generation
Rebuffel, C., Roberti, M., Soulier, L., Scoutheeten, G., Cancelliere, R., & Gallinari, P. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-021-00801-4 · Cited by 50 · see citing works
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark
Dziri, N., Rashkin, H., Linzen, T., & Reitter, D. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00506 · Cited by 67 · see citing works
AI Alignment: A Contemporary Survey
Ji, J.-M., Qiu, T., Chen, B.-Y., Zhou, J.-Y., Zhang, B.-R., Hong, D., Lou, H., Wang, K.-L., Duan, Y., He, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3770749 · Cited by 57 · see citing works
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems
Thomson, C., & Reiter, E. (2020) · Proceedings of the 13th International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.22 · Cited by 38 · see citing works
Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning
Wang, F., Xu, Z., Szekely, P. A., & Chen, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.371 · Cited by 38 · see citing works
Focus Attention: Promoting Faithfulness and Diversity in Summarization
Aralikatte, R., Narayan, S., Maynez, J., Rothe, S., & McDonald, R. (2021) · doi:10.18653/v1/2021.acl-long.474 · Cited by 38 · see citing works
Perturbation CheckLists for Evaluating NLG Evaluation Metrics
Sai, A. B., Dixit, T., Sheth, D. Y., Mohan, S., & Khapra, M. M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.575 · Cited by 34 · see citing works
Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation
Rebuffel, C., Scialom, T., Soulier, L., Piwowarski, B., Lamprier, S., Staiano, J., Scoutheeten, G., & Gallinari, P. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.633 · Cited by 24 · see citing works
A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generation
Narayan, S., Simões, G., Zhao, Y., Maynez, J., Das, D., Collins, M., & Lapata, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.94 · Cited by 25 · see citing works
Self-citationWikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Chen, M., Wiseman, S., & Gimpel, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.17 · Cited by 25 · see citing works
ENT-DESC: Entity Description Generation by Exploring Knowledge Graph
Cheng, L., Wu, D., Bing, L., Zhang, Y., Jie, Z., Lu, W., & Si, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.90 · Cited by 22 · see citing works
A Survey on Neural Data-to-Text Generation
Lin, Y., Ruan, T., Liu, J., & Wang, H. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3304385 · Cited by 22 · see citing works
Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View
Liu, T., Zheng, X., Chang, B., & Sui, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17583 · Cited by 21 · see citing works
Unifying Structured Data as Graph for Data-to-Text Pre-Training
Li, S., Li, L., Geng, R., Yang, M., Li, B., Yuan, G., He, W., Yuan, S. F., Ma, C., Huang, F., et al. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00641 · Cited by 18 · see citing works
Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story GenerationPreprint
Chhun, C., Colombo, P., Clavel, C., & Suchanek, F. M. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.11646 · Cited by 29 · see citing works
ChatGPT Incorrectness Detection in Software Reviews
Tanzil, M. H., Khan, J. Y., & Uddin, G. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639194 · Cited by 12 · see citing works
Stage-wise Fine-tuning for Graph-to-Text Generation
Wang, Q., Yavuz, S., Lin, X., Ji, H., & Rajani, N. F. (2021) · doi:10.18653/v1/2021.acl-srw.2 · Cited by 14 · see citing works
Search and Learn: Improving Semantic Coverage for Data-to-Text Generation
Jolly, S., Zhang, Z. X., Dengel, A. R., & Mou, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21332 · Cited by 13 · see citing works
TWT: Table with Written Text for Controlled Data-to-Text Generation
Li, T., Fang, L., Lou, J., & Li, Z. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.107 · Cited by 12 · see citing works
Attend, Memorize and Generate: Towards Faithful Table-to-Text Generation in Few Shots
Zhao, W., Ye, L., Wan, Y., & Yu, P. S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.347 · Cited by 12 · see citing works
R2D2: Robust Data-to-Text with Replacement Detection
Nan, L., Flores, L. J., Zhao, Y., Liu, Y., Benson, L., Zou, W., & Radev, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.464 · Cited by 14 · see citing works
FaMeSumm: Investigating and Improving Faithfulness of Medical Summarization
Zhang, N., Zhang, Y., Guo, W., Mitra, P., & Zhang, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.673 · Cited by 15 · see citing works
Generative adversarial network for Table-to-Text generation
Zhao, J., Zhan, Z., Li, T., Li, R., Hu, C., Wang, S., & Zhang, Y. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.04.036 · Cited by 11 · see citing works
(2021) · doi:10.18653/v1/2021.findings-acl.427 · Cited by 10 · see citing works
Truth-Conditional Captions for Time Series Data
Jhamtani, H., & Berg-Kirkpatrick, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.55 · Cited by 10 · see citing works
Montella, S., Fabre, B., Urvoy, T., Heinecke, J., & Rojas-Barahona, L. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00571 · Cited by 12 · see citing works
Data-to-text Generation by Splicing Together Nearest Neighbors
Wiseman, S., Bačkurs, A., & Stratos, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.352 · Cited by 10 · see citing works
Generation-Focused Table-Based Intermediate Pre-training for Free-Form Question Answering
Shi, P. J., Ng, P., Nan, F., Zhu, H., Wang, J., Jiang, J., Li, A. H., Chakravarti, R., Weidner, D., Xiang, B., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21382 · Cited by 10 · see citing works
Yu, L., Alégroth, E., Chatzipetrou, P., & Gorschek, T. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107802 · Cited by 8 · see citing works
SynthBio: A Case Study in Human-AI Collaborative Curation of Text DatasetsPreprint
Yuan, A., Ippolito, D., Nikolaev, V., Callison-Burch, C., Coenen, A., & Gehrmann, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.06467 · Cited by 9 · see citing works
MedT2T: An adaptive pointer constrain generating method for a new medical text-to-table task
Zhao, W., Gu, D., Yang, X., Jia, M., Liang, C., Wang, X., & Золотарев, О. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.07.030 · Cited by 7 · see citing works
semiPQA: A Study on Product Question Answering over Semi-structured Data
Shen, X., Barlacchi, G., Tredici, M. D., Cheng, W., & de Gispert, A. (2022) · Workshop on e-Commerce and NLP (ECNLP) · doi:10.18653/v1/2022.ecnlp-1.14 · Cited by 8 · see citing works
Benchmarking and Assessing Transformations Between Text and Causal Maps via Large Language Models
Giabbanelli, P. J., Gandee, T. J., Agrawal, A., & Hosseinichimeh, N. (2025) · Applied Ontology · doi:10.1177/15705838241304102 · Cited by 8 · see citing works
Entity-Based Semantic Adequacy for Data-to-Text Generation
Faille, J., Gatt, A., & Gardent, C. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.findings-emnlp.132 · Cited by 11 · see citing works
XWikiGen: Cross-lingual Summarization for Encyclopedic Text Generation in Low Resource Languages
Taunk, D., Sagare, S., Patil, A., Subramanian, S., Gupta, M., & Varma, V. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583405 · Cited by 7 · see citing works
TaTA: A Multilingual Table-to-Text Dataset for African Languages
Gehrmann, S., Ruder, S., Nikolaev, V. A., Botha, J., Chavinda, M., Parikh, A. P., & Rivera, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.118 · Cited by 7 · see citing works
Self-citationNeural Methods for Data-to-text Generation
Sharma, M., Gogineni, A. K., & Ramakrishnan, N. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3660639 · Cited by 17 · see citing works
A Tutorial on Evaluation Metrics used in Natural Language Generation
Khapra, M. M., & Sai, A. B. (2021) · doi:10.18653/v1/2021.naacl-tutorials.4 · Cited by 7 · see citing works
ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models
Xiang, J., Liu, Z., Zhou, Y., Xing, E. P., & Hu, Z. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.136 · Cited by 7 · see citing works
Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation
Huang, F., Ke, P., & Huang, M. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00582 · Cited by 6 · see citing works
Localize, Retrieve and Fuse: A Generalized Framework for Free-Form Question Answering over Tables
Zhao, W., Liu, Y., Wan, Y., Wang, Y., Deng, Z., & Yu, P. S. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.1 · Cited by 7 · see citing works
Enhancing consistency with the fusion of paralleled decoders for text generation
Li, Y., Huang, H., Bai, Y., & Gao, Y. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102652 · Cited by 7 · see citing works
Language Model Augmented Relevance Score
Liu, R., Wei, J., & Vosoughi, S. (2021) · doi:10.18653/v1/2021.acl-long.521 · Cited by 6 · see citing works
Semantic Verification in Large Language Model-based Retrieval Augmented Generation
Martin, A., Witschel, H. F., Mandl, M. M., & Stockhecke, M. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31199 · Cited by 7 · see citing works
Bootstrapping Generators from Noisy Data
Perez-Beltrachini, L., & Lapata, M. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1137 · Cited by 6 · see citing works
Rebuffel, C., Soulier, L., Scoutheeten, G., & Gallinari, P. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.18 · Cited by 6 · see citing works
MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization
Xu, X., Dušek, O., Narayan, S., Rieser, V., & Konstas, I. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.133 · Cited by 7 · see citing works
Faithful Low-Resource Data-to-Text Generation through Cycle Training
Wang, Z., Collins, M., Vedula, N., Filice, S., Malmasi, S., & Rokhlenko, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.160 · Cited by 5 · see citing works
FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation
Goyal, T., Li, J. J., & Durrett, G. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.35 · Cited by 5 · see citing works
CogNLG: Cognitive graph for KG‐to‐text generation
Lai, P., Ye, F., Fu, Y.-G., Chen, Z., Wu, Y., Wang, Y., & Chang, V. I. C. (2023) · Expert Systems · doi:10.1111/exsy.13461 · Cited by 5 · see citing works
FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation
Zhang, K., Balalau, O., & Manolescu, I. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.672 · Cited by 5 · see citing works
Table-to-Text Generation With Pretrained Diffusion Models
Krylov, A. S., & Somov, O. (2024) · IEEE Access · doi:10.1109/access.2024.3440006 · Cited by 5 · see citing works
Semantic Matching for Sequence-to-Sequence Learning
Zhang, R., Chen, C., Zhang, X., Bai, K., & Carin, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.21 · Cited by 5 · see citing works
Precisely the Point: Adversarial Augmentations for Faithful and Informative Text Generation
Wu, W., Li, W., Liu, J., Xiao, X., Li, S., & Lyu, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.482 · Cited by 6 · see citing works
XF2T: Cross-lingual Fact-to-Text Generation for Low-Resource Languages
Sagare, S., Abhishek, T., Singh, B., Sharma, A., Gupta, M., & Varma, V. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.2 · Cited by 5 · see citing works
Time-aware Prompting for Text Generation
Cao, S., & Wang, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.535 · Cited by 4 · see citing works
Enhancing diversity for logical table‐to‐text generation with mixture of experts
Wu, J., & Hou, M. (2024) · Expert Systems · doi:10.1111/exsy.13533 · Cited by 4 · see citing works
Scrutinizing the foundations: could large language models be solipsistic?
Esanu, A. (2024) · Synthese · doi:10.1007/s11229-024-04589-w · Cited by 4 · see citing works
OpenRT: An Open-source Framework for Reasoning Over Tabular Data
Zhao, Y., Mi, B., Qi, Z., Nan, L., Guo, M., Cohan, A., & Radev, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.32 · Cited by 5 · see citing works
Few-Shot Data-to-Text Generation via Unified Representation and Multi-Source Learning
Li, A. H., Shang, M., Spiliopoulou, E., Ma, J., Ng, P., Wang, Z., Min, B., Wang, W. Y., McKeown, K. R., Castelli, V., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.894 · Cited by 4 · see citing works
Active Evaluation: Efficient NLG Evaluation with Few Pairwise Comparisons
Mohankumar, A. K., & Khapra, M. M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.600 · Cited by 4 · see citing works
Text-in-Context: Token-Level Error Detection for Table-to-Text Generation
Kasner, Z., Mille, S., & Dušek, O. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.25 · Cited by 3 · see citing works
Fact Checking Machine Generated Text with Dependency Trees
Estes, A., Vedula, N., Collins, M., Cecil, M., & Rokhlenko, O. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.46 · Cited by 3 · see citing works
Unsupervised Token-level Hallucination Detection from Summary Generation By-products
Marfurt, A., & Henderson, J. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.21 · Cited by 5 · see citing works
Plan-then-Seam: Towards Efficient Table-to-Text Generation
Li, L., Geng, R., Fang, C., Li, B., Ma, C., Li, B., & Li, Y. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.15 · Cited by 4 · see citing works
ReTAG: Reasoning Aware Table to Analytic Text Generation
Ghosal, D., Nema, P., & Raghuveer, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.389 · Cited by 4 · see citing works
Adapting Knowledge for Few-Shot Table-to-Text Generation
Guo, Z., Yan, M., Qi, J., Zhou, J., He, Z., Zheng, G., Wang, X., & Zhou, C. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3430480 · Cited by 5 · see citing works
Trainable Ranking Models to Evaluate the Semantic Accuracy of Data-to-Text Neural Generator
Garneau, N., & Lamontagne, L. (2021) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2021.eval4nlp-1.6 · Cited by 3 · see citing works
Lan, Z., Li, W., Su, J., Xiao, X., Liu, J., Wu, W., & Lyu, Y. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14267 · Cited by 3 · see citing works
Plan-then-Generate: Controlled Data-to-Text Generation via PlanningPreprint
Su, Y., Vandyke, D., Wang, S., Fang, Y., & Collier, N. (2021) · arXiv (Cornell University) · doi:10.17863/cam.84984 · Cited by 7 · see citing works
A comprehensive study on factual consistency in LLMs for data-to-text generation
Mahapatra, J., & Garain, U. (2025) · Sadhana · doi:10.1007/s12046-025-02860-5 · Cited by 4 · see citing works
A framework to synthetically generate fine-grained hallucinated data
Saxena, A., Aggarwal, D., Badathala, N., & Bhattacharyya, P. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09864-x · Cited by 4 · see citing works
- Authors
- Dhingra, B., Faruqui, M., Parikh, A., Chang, M.-W., Das, D., & Cohen, W.
- Affiliations
- Google (United States) (US)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 4884-4895
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2948628052
- Cited by
- 150
- References
- 40
- Citations by year
- 2026: 7 · 2025: 17 · 2024: 13 · 2023: 33 · 2022: 26 · 2021: 35 · 2020: 17 · 2019: 1 · 2018: 1
- FWCI
- 7.56
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- table-to-text generation, evaluation metrics, semi-structured data, human evaluation correlation, WebNLG, BLEU, ROUGE
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
