CoChat
Sign Up
Cited by 21Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View

Open access
Liu, Tianyu, Zheng, Xin, Chang, Baobao, Sui, Zhifang·2021·Proceedings of the AAAI Conference on Artificial Intelligence
DOI 10.1609/aaai.v35i15.17583
Open this paper in CoChat — free account

Abstract

In open domain table-to-text generation, we notice the unfaithful generation usually contains hallucinated entities which can not be aligned to any input table record. We thus try to evaluate the generation faithfulness with two entity-centric metrics: table record coverage and the ratio of hallucinated entities in text, both of which are shown to have strong agreement with human judgements. Then based on these metrics, we quantitatively analyze the correlation between training data quality and generation fidelity which indicates the potential usage of entity information in faithful generation. Motivated by these findings, we propose two methods for faithful generation: 1) augmented training by incorporating the auxiliary entity information, including both an augmented plan-based model and an unsupervised model and 2) training instance selection based on faithfulness ranking. We show these approaches improve generation fidelity in both full dataset setting and few shot setting by both automatic and human evaluations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1609/aaai.v35i15.17583

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Planning and skeleton generation are common methods to improve faithfulness to the input in data-to-text tasks.

    “Planning and skeleton generation are common methods to improve the faithfulness to the input in data-to-text tasks.”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [162]

    The Introduction (p. 1) says two-stage plan-based generation can improve accuracy and describes the paper’s augmented plans as reducing hallucinations. Related Work (p. 7) also identifies planning as an established approach in data-to-text generation. Our copy does not identify skeleton generation as a common method for improving faithfulness, so that part of the claim is unsupported. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 56 · Cited by 21

Research works printed but not linked: 50

  1. Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR abs/1409.0473. URL http://arxiv.org/abs/1409.0473. · arXiv
  2. Bao, J.; Tang, D.; Duan, N.; Yan, Z.; Lv, Y.; Zhou, M.; and Zhao, T. 2018. Table-to-Text: Describing Table Region With Natural Language. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI18), New Orleans, Louisiana, USA, February 2-7, 2018 , 50205027. URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/ paper/view/16138.
  3. Barzilay, R.; and Lapata, M. 2005. Collective content selection for concept-to-text generation. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing , 331-338. Association for Computational Linguistics.
  4. Barzilay, R.; and Lapata, M. 2006. Aggregation via set partitioning for natural language generation. In NAACL , 359-366. Association for Computational Linguistics.
  5. Belz, A. 2008. Automatic generation of weather forecast texts using comprehensive probabilistic generation-space models. Natural Language Engineering 14(4): 431-455.
  6. Chen, Y.-C.; and Bansal, M. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. arXiv preprint arXiv:1805.11080 . · arXiv
  7. Chisholm, A.; Radford, W.; and Hachey, B. 2017. Learning to generate one-sentence biographies from wikidata. arXiv preprint arXiv:1702.06235 . · arXiv
  8. Colin, E.; Gardent, C.; M'rabet, Y.; Narayan, S.; and PerezBeltrachini, L. 2016. The webnlg challenge: Generating text from dbpedia data. In Proceedings of the 9th International Natural Language Generation conference , 163-167.
  9. Duboue, P. A.; and McKeown, K. R. 2002. Content planner construction via evolutionary algorithms and a corpus-based fitness function. In Proceedings of INLG 2002 , 89-96.
  10. Duˇ sek, O.; Novikova, J.; and Rieser, V . 2019. Evaluating the Stateof-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge. arXiv preprint arXiv:1901.11528 URL https:// arxiv.org/abs/1901.11528. · arXiv
  11. Eyal, M.; Baumel, T.; and Elhadad, M. 2019. Question answering as an automatic evaluation metric for news article summarization. arXiv preprint arXiv:1906.00318 . · arXiv
  12. Falke, T.; Ribeiro, L. F.; Utama, P. A.; Dagan, I.; and Gurevych, I. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2214-2220.
  13. Fan, A.; Lewis, M.; and Dauphin, Y. 2019. Strategies for structuring story generation. arXiv preprint arXiv:1902.01109 . · arXiv
  14. Ferreira, T. C.; van der Lee, C.; van Miltenburg, E.; and Krahmer, E. 2019. Neural data-to-text generation: A comparison between pipeline and end-to-end architectures. arXiv preprint arXiv:1908.09022 . · arXiv
  15. Goodrich, B.; Rao, V.; Liu, P. J.; and Saleh, M. 2019. Assessing the factual accuracy of generated text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &Data Mining , 166-175.
  16. Jhamtani, H.; Gangal, V.; Hovy, E.; Neubig, G.; and BergKirkpatrick, T. 2018. Learning to generate move-by-move commentary for chess games from large-scale social forum data. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , volume 1, 1661-1671.
  17. Kaffee, L.-A.; Elsahar, H.; Vougiouklis, P.; Gravier, C.; Laforest, F.; Hare, J.; and Simperl, E. 2018. Learning to generate wikipedia summaries for underserved languages from wikidata. arXiv preprint arXiv:1803.07116 . · arXiv
  18. Kim, J.; and Mooney, R. J. 2010. Generative alignment and semantic parsing for learning from ambiguous supervision. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters , 543-551. Association for Computational Linguistics.
  19. Konstas, I.; and Lapata, M. 2013. A global model for conceptto-text generation. Journal of Artificial Intelligence Research 48: 305-346.
  20. Laha, A.; Jain, P.; Mishra, A.; and Sankaranarayanan, K. 2018. Scalable Micro-planned Generation of Discourse from Structured Data. CoRR abs/1810.02889. URL http://arxiv.org/abs/1810. 02889. · arXiv
  21. Lebret, R.; Grangier, D.; and Auli, M. 2016. Neural text generation from structured data with application to the biography domain. arXiv preprint arXiv:1603.07771 . · arXiv
  22. Li, X. L.; and Rush, A. M. 2020. Posterior Control of Blackbox Generation. arXiv preprint arXiv:2005.04560 . · arXiv
  23. Liang, P.; Jordan, M. I.; and Klein, D. 2009. Learning semantic correspondences with less supervision. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1 , 91-99. Association for Computational Linguistics.
  24. Liu, T.; Luo, F.; Xia, Q.; Ma, S.; Chang, B.; and Sui, Z. 2019a. Hierarchical encoder with auxiliary supervision for neural tableto-text generation: Learning better representation for tables. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, 6786-6793.
  25. Liu, T.; Luo, F.; Yang, P.; Wu, W.; Chang, B.; and Sui, Z. 2019b. Towards comprehensive description generation from factual attribute-value tables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 5985-5996.
  26. Liu, T.; Wang, K.; Sha, L.; Chang, B.; and Sui, Z. 2018. Tableto-Text Generation by Structure-Aware Seq2seq Learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , 4881-4888. URL https: //www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16599.
  27. Liu, T.; Wei, B.; Chang, B.; and Sui, Z. 2017. Large-scale simple question generation by template-based seq2seq learning. In National CCF Conference on Natural Language Processing and Chinese Computing , 75-87. Springer.
  28. Ma, S.; Yang, P.; Liu, T.; Li, P.; Zhou, J.; and Sun, X. 2019. Key Fact as Pivot: A Two-Stage Model for Low Resource Table-to-Text Generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , 2047-2057. Florence, Italy: Association for Computational Linguistics. doi:10.18653/v1/P191197. URL https://www.aclweb.org/anthology/P19-1197. · DOI
  29. Mei, H.; Bansal, M.; and Walter, M. R. 2016. What to talk about and how? Selective Generation using LSTMs with Coarse-to-Fine Alignment. In NAACL HLT 2016 , 720-730. URL http://aclweb. org/anthology/N/N16/N16-1086.pdf.
  30. Moryossef, A.; Dagan, I.; and Goldberg, Y. 2019. Improving Quality and Efficiency in Plan-based Neural Data-to-Text Generation. arXiv preprint arXiv:1909.09986 . · arXiv
  31. Moryossef, A.; Goldberg, Y.; and Dagan, I. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. arXiv preprint arXiv:1904.03396 . · arXiv
  32. Ott, M.; Edunov, S.; Baevski, A.; Fan, A.; Gross, S.; Ng, N.; Grangier, D.; and Auli, M. 2019. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038 . · arXiv
  33. Parikh, A. P.; Wang, X.; Gehrmann, S.; Faruqui, M.; Dhingra, B.; Yang, D.; and Das, D. 2020. ToTTo: A Controlled Table-To-Text Generation Dataset. arXiv preprint arXiv:2004.14373 . · arXiv
  34. Peng, N.; Ghazvininejad, M.; May, J.; and Knight, K. 2018. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling , 43-49.
  35. Perez-Beltrachini, L.; and Lapata, M. 2018. Bootstrapping generators from noisy data. arXiv preprint arXiv:1804.06385 . · arXiv
  36. Puduppully, R.; Dong, L.; and Lapata, M. 2018. Data-to-Text Generation with Content Selection and Planning. arXiv preprint arXiv:1809.00582 . · arXiv
  37. Qi, P.; Zhang, Y.; Zhang, Y.; Bolton, J.; and Manning, C. D. 2020. Stanza: A Python Natural Language Processing Toolkit for Many Human Languages. arXiv preprint arXiv:2003.07082 . · arXiv
  38. Reiter, E.; and Dale, R. 1997. Building applied natural language generation systems. Natural Language Engineering 3(1): 57-87.
  39. Scialom, T.; Lamprier, S.; Piwowarski, B.; and Staiano, J. 2019. Answers unite! unsupervised metrics for reinforced summarization models. arXiv preprint arXiv:1909.01610 . · arXiv
  40. See, A.; Liu, P. J.; and Manning, C. D. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368 . · arXiv
  41. Sha, L.; Mou, L.; Liu, T.; Poupart, P.; Li, S.; Chang, B.; and Sui, Z. 2018. Order-Planning Neural Text Generation From Structured Data. In McIlraith, S. A.; and Weinberger, K. Q., eds., Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018 , 5414-5421. AAAI Press. URL https://www.aaai.org/ocs/index.php/AAAI/AAAI18/ paper/view/16203.
  42. Shen, X.; Chang, E.; Su, H.; Zhou, J.; and Klakow, D. 2020. Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence. arXiv preprint arXiv:2005.01096 . · arXiv
  43. Shen, X.; Suzuki, J.; Inui, K.; Su, H.; Klakow, D.; and Sekine, S. 2019. Select and Attend: Towards Controllable Content Selection in Text Generation. arXiv preprint arXiv:1909.04453 . · arXiv
  44. Tian, R.; Narayan, S.; Sellam, T.; and Parikh, A. P. 2019. Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation. arXiv preprint arXiv:1910.08684 . · arXiv
  45. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems , 5998-6008.
  46. Wang, A.; Cho, K.; and Lewis, M. 2020. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. arXiv preprint arXiv:2004.04228 . · arXiv
  47. Wang, K.; Liu, T.; Chang, B.; and Sui, Z. 2020a. An AnchorBased Automatic Evaluation Metric for Document Summarization. In Proceedings of the 28th International Conference on Computational Linguistics , 5696-5701.
  48. Wang, Q.; Pan, X.; Huang, L.; Zhang, B.; Jiang, Z.; Ji, H.; and Knight, K. 2018. Describing a knowledge base. arXiv preprint arXiv:1809.01797 . · arXiv
  49. Wiseman, S.; Shieber, S. M.; and Rush, A. M. 2018. Learning neural templates for text generation. arXiv preprint arXiv:1808.10122 . · arXiv
  50. Ye, R.; Shi, W.; Zhou, H.; Wei, Z.; and Li, L. 2020. Variational Template Machine for Data-to-Text Generation. arXiv preprint arXiv:2002.01127 . · arXiv

Authors
Liu, T., Zheng, X., Chang, B., & Sui, Z.
Affiliations
  • Peking University (CN)
  • Chinese Academy of Sciences (CN)
ORCID
Venue
Proceedings of the AAAI Conference on Artificial Intelligence
Publisher
Association for the Advancement of Artificial Intelligence (AAAI)
Published
2021-05-18
Volume
35
Issue
15
Pages
13415-13423
ISSN
  • 2159-5399 (ISSN-L)
  • 2374-3468
DOI
10.1609/aaai.v35i15.17583
OpenAlex
W3175929660
Cited by
21
References
56
Citations by year
2025: 2 · 2024: 4 · 2023: 5 · 2022: 6 · 2021: 4
FWCI
2.19
Type
conference paper
Language
en
Open access
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (61876004, U19A2065)
  • National Key Research and Development Program of China
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
table-to-text generation, open-domain question answering, hallucinations, training data quality, data augmentation, instance selection, few-shot learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review