A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems
Open accessAbstract
Most Natural Language Generation systems need to produce accurate texts.We propose a methodology for high-quality human evaluation of the accuracy of generated texts, which is intended to serve as a gold-standard for accuracy evaluations of data-to-text systems.We use our methodology to evaluate the accuracy of computer generated basketball summaries.We then show how our gold standard evaluation can be used to validate automated metrics.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Extrinsic hallucination is not always erroneous because it could be from factually correct external information.
“Notably, the extrinsic hallucination is not always erroneous because it could be from factually correct external information [177, 247].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [247]
The tolerance of hallucination is very low when data-to-text generation is applied to the real world, such as in patient information table description and analysis of experimental results tables in a scientific report.
“Moreover, the tolerance of hallucination is very low when this task is applied to the real world, such as in the case of patient information table description [247], and analysis of experimental results tables in a scientific report.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [247]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · see citing works
FEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · see citing works
Evaluating Content Selection in Summarization: The Pyramid Method
Nenkova, A., & Passonneau, R. (2004) · Columbia University · doi:10.7916/d80r9xvd · Cited by 527 · see citing works
Challenges in Data-to-Document Generation
Wiseman, S., Shieber, S., & Rush, A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1239 · Cited by 524 · see citing works
A Structured Review of the Validity of BLEU
Reiter, E. (2018) · Computational Linguistics · doi:10.1162/coli_a_00322 · Cited by 386 · see citing works
Self-citationChoosing words in computer-generated weather forecasts
Reiter, E., Sripada, S. G., Jim Hunter, A., Yu, J. S., & Davy, I. P. (2005) · Artificial Intelligence · doi:10.1016/j.artint.2005.06.006 · Cited by 262 · see citing works
Self-citationThe Fact Extraction and VERification (FEVER) Shared Task
Thorne, J., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the First Workshop on Fact Extraction and VERification (FEVER) · doi:10.18653/v1/w18-5501 · Cited by 212 · see citing works
Evaluation of Text Generation: A Survey
Celikyilmaz, A., Clark, E., & Gao, J. (2020) · arXiv · doi:10.48550/arxiv.2006.14799 · Cited by 207 · see citing works
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics
Mathur, N., Baldwin, T., & Cohn, T. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.448 · Cited by 206 · see citing works
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205 · cited 1× · see citing works
“Human evaluation of NLG systems is usually done using Likert scales or ratings (van der Lee et al., 2019).”
MethodReiter, E., & Belz, A. (2009) · Computational Linguistics · doi:10.1162/coli.2009.35.4.35405 · Cited by 204 · see citing works
Self-citationEvaluating the state-of-the-art of End-to-End Natural Language Generation: The E2E NLG challenge
Dušek, O., Novikova, J., & Rieser, V. (2019) · Computer Speech & Language · doi:10.1016/j.csl.2019.06.009 · Cited by 194 · see citing works
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Dhingra, B., Faruqui, M., Parikh, A., Chang, M.-W., Das, D., & Cohen, W. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1483 · Cited by 150 · see citing works
Findings of the E2E NLG Challenge
Dušek, O., Novikova, J., & Rieser, V. (2018) · International Conference on Natural Language Generation · doi:10.18653/v1/w18-6539 · Cited by 95 · see citing works
Semantic Noise Matters for Neural Natural Language Generation
Dušek, O., Howcroft, D. M., & Rieser, V. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8652 · Cited by 85 · see citing works
RankME: Reliable Human Ratings for Natural Language Generation
Novikova, J., Dušek, O., & Rieser, V. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2012 · Cited by 83 · see citing works
Thorne, J. P., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., & Mittal, A. (2019) · Fact Extraction and VERification Workshop (FEVER) · doi:10.18653/v1/d19-6601 · Cited by 68 · see citing works
Gong, H., Feng, X.-C., Qin, B., & Liu, T. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d19-1310 · Cited by 65 · see citing works
A Hierarchical Model for Data-to-Text Generation
Rebuffel, C., Soulier, L., Scoutheeten, G., & Gallinari, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5_5 · Cited by 52 · see citing works
PEAK: Pyramid Evaluation via Automated Knowledge Extraction
Yang, Q., Passonneau, R. J., & de Melo, G. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10336 · Cited by 51 · see citing works
Revisiting Challenges in Data-to-Text Generation with Fact Grounding
Wang, H. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8639 · Cited by 34 · see citing works
(2019) · doi:10.1609/aaai.v33i01.33016908 · Cited by 26 · see citing works
Shared Task on Evaluating Accuracy
Reiter, E., & Thomson, C. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.28 · Cited by 8 · see citing works
Self-citationStudying the Impact of Filling Information Gaps on the Output Quality of Neural Data-to-Text
Thomson, C., Zhao, Z., & Sripada, S. G. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.6 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Ani Nenkova and Rebecca Passonneau. 2004. Evaluat- ing content selection in summarization: The pyra- mid method. In Proceedings of the Human Lan- guage Technology Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152, Boston, Massachusetts, USA. Association for Com- putational Linguistics.
- Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with content selection and planning. In Proceedings of the 33rd AAAI Confer- ence on Artificial Intelligence, Honolulu, Hawaii.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation
Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., & Macherey, W. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00437 · Cited by 271 · see citing works
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text
Gehrmann, S., Clark, E., & Sellam, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13715 · Cited by 123 · see citing works
Underreporting of errors in NLG output, and what to do about it
van Miltenburg, E., Clinciu, M., Dušek, O., Gkatzia, D., Inglis, S., Leppänen, L., Mahamood, S., Manning, E., Schoch, S., Thomson, C., et al. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.14 · Cited by 26 · see citing works
Self-citationA New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection
Yang, S., Sun, R., & Wan, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.256 · Cited by 32 · see citing works
Beyond Chatbot-K: On Large Language Models, “Generative AI,” and Rise of Chatbots—An Introduction
Goodlad, L. M. E., & Stone, M. (2024) · Critical AI · doi:10.1215/2834703x-11205147 · Cited by 31 · see citing works
Corbelle, J. G., Bugarín, A., Alonso, J. M., & Taboada, J. J. (2022) · International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.10 · Cited by 25 · see citing works
Santhanam, S., Hedayatnia, B., Gella, S., Padmakumar, A., Kim, S., Liu, Y., & Hakkani-Tur, D. (2021) · arXiv · doi:10.48550/arxiv.2110.05456 · Cited by 21 · see citing works
Data-to-text Generation with Variational Sequential Planning
Puduppully, R., Fu, Y., & Lapata, M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00484 · Cited by 19 · see citing works
How Do Seq2Seq Models Perform on End-to-End Data-to-Text Generation?
Yin, X., & Wan, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.531 · Cited by 20 · see citing works
Generation Challenges: Results of the Accuracy Evaluation Shared Task
Thomson, C., & Reiter, E. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.23 · Cited by 14 · see citing works
Self-citationThe Pitfalls of Defining Hallucination
Deemter, K. V. (2024) · Computational Linguistics · doi:10.1162/coli_a_00509 · Cited by 13 · see citing works
Evaluating factual accuracy in complex data-to-text
Thomson, C., Reiter, E., & Sundararajan, B. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101482 · Cited by 12 · see citing works
Self-citationEvaluation of Question Answering Systems: Complexity of Judging a Natural Language
Farea, A., Yang, Z., Duong, K., Perera, N., & Emmert‐Streib, F. (2025) · ACM Computing Surveys · doi:10.1145/3744663 · Cited by 21 · see citing works
Fan, A., & Gardent, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.586 · Cited by 18 · see citing works
Popović, M. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.18 · Cited by 9 · see citing works
Xue, B., Wang, W., Wang, H., Mi, F., Wang, R., Wang, Y., Shang, L., Jiang, X., Liu, Q., & Wong, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.525 · Cited by 11 · see citing works
Das, A. B., Ahmed, S., & Sakib, S. K. (2025) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi67747.2025.11269492 · Cited by 10 · see citing works
Tackling Hallucinations in Neural Chart Summarization
Islam, S. O. U., Škrjanec, I., Dušek, O., & Demberg, V. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.30 · Cited by 6 · see citing works
Barriers and enabling factors for error analysis in NLG research
van Miltenburg, E., Clinciu, M., Dušek, O., Gkatzia, D., Inglis, S., Leppänen, L., Mahamood, S., Schoch, S., Thomson, C., & Wen, L. (2023) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.2023.4529 · Cited by 5 · see citing works
Self-citationComparing informativeness of an NLG chatbot vs graphical app in diet-information domain
Balloccu, S., & Reiter, E. (2022) · International Conference on Natural Language Generation · doi:10.18653/v1/2022.inlg-main.13 · Cited by 4 · see citing works
Self-citationHistAlign: Improving Context Dependency in Language Generation by Aligning with History
Wan, D., Zhang, S., & Bansal, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.179 · Cited by 5 · see citing works
Text-in-Context: Token-Level Error Detection for Table-to-Text Generation
Kasner, Z., Mille, S., & Dušek, O. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.25 · Cited by 3 · see citing works
Towards a Consensus Taxonomy for Annotating Errors in Automatically Generated Text
Huidrom, R., & Belz, A. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_058 · Cited by 3 · see citing works
On reporting scores and agreement for error annotation tasks
Popović, M., & Belz, A. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.26 · Cited by 3 · see citing works
Automatic Generation of Text for Match Recaps using Esport Caster Commentaries
Olarewaju, O., Kokkinakis, A. V., Demediuk, S., Roberstson, J., Nölle, I., Patra, S., Slawson, D., Chitayat, A. P., Coates, A., Kirman, B. J., et al. (2020) · Computer Science & Information Technology (CS & IT) · doi:10.5121/csit.2020.101810 · Cited by 3 · see citing works
Shared Task in Evaluating Accuracy: Leveraging Pre-Annotations in the Validation Process
Garneau, N., & Lamontagne, L. (2021) · International Conference on Natural Language Generation · doi:10.18653/v1/2021.inlg-1.26 · Cited by 3 · see citing works
Error Analysis of ToTTo Table-to-Text Neural NLG Models
Sundararajan, B., Sripada, S. G., & Reiter, E. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.43 · Cited by 2 · see citing works
Self-citationTowards In-Context Non-Expert Evaluation of Reflection Generation for Counselling Conversations
Wu, Z., Balloccu, S., Helaoui, R., Recupero, D. R., & Riboni, D. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.9 · Cited by 2 · see citing works
Berro, A., Benatallah, B., Gaci, Y., & Benabdeslem, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70341-6_20 · Cited by 1 · see citing works
Shared Task on Evaluating Accuracy in Natural Language GenerationPreprint
Reiter, E., & Thomson, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.12234
Self-citationUm agente de jornalismo automatizado cobrindo a Amazônia Azul.
Sym, Y. V. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/d.3.2023.tde-22082023-142314
Reiter, E. (2024) · Natural Language Generation · doi:10.1007/978-3-031-68582-8_4
Self-citationReiter, E. (2024) · Natural Language Generation · doi:10.1007/978-3-031-68582-8_5
Self-citationText Generation with and without Retrieval
Fan, H. (2021) · theses.fr (ABES) · doi:10.70675/87ca97d1z017cz478azac26z38a64f0a75e0
Controllable and Document-Level Text Simplification
Cripwell, L. (2023) · theses.fr (ABES) · doi:10.70675/39a81752z69b3z4c9bzb9f9z4ac49474d4c5
Data-Based Natural Language Generation : Evaluation and Explainability
Faille, J. (2023) · theses.fr (ABES) · doi:10.70675/3e73dd43z7aa6z4405za5cez27c78b408a3c
ErrorBench: Fine-Grained Error Analysis of Multi-Family LLMs in Data-to-Text Generation
Bharadwaj, S., & Anand, A. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728592
- Authors
- Thomson, C., & Reiter, E.
- Affiliations
- University of Aberdeen (GB)
- ORCID
- Reiter, Ehud
- Venue
- Proceedings of the 13th International Conference on Natural Language Generation
- Published
- 2020
- Pages
- 158-168
- OpenAlex
- W3102266532
- Cited by
- 38
- References
- 21
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 5 · 2023: 11 · 2022: 8 · 2021: 8 · 2020: 3
- FWCI
- 2.86
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- International Conference on Natural Language Generation (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Aberdeen University Research Archive (Aberdeen University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- natural language generation, accuracy assessment, human evaluation, automatic evaluation metrics, metric validation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
