CoChat
Sign Up
Cited by 252Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

ToTTo: A Controlled Table-To-Text Generation Dataset

Open access
Parikh, Ankur, Wang, Xuezhi, Gehrmann, Sebastian, Faruqui, Manaal, Dhingra, Bhuwan, Yang, Diyi, Das, Dipanjan·2020·Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
DOI 10.18653/v1/2020.emnlp-main.89
Open this paper in CoChat — free accountOpen PDF

Abstract

Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, Dipanjan Das. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2020.emnlp-main.89

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • An encoder with a defective comprehension ability could influence the degree of hallucination.

    “An encoder with a defective comprehension ability could influence the degree of hallucination [198].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [198]

    The Hallucination section reports that models produce content unfaithful to the table and struggle with faithfulness even when references are faithful. The Diverse table structure and numerical reasoning section says the model cannot make some table-based inferences from its source representation. Neither passage identifies defective encoder comprehension as the cause or measures its effect on the degree of hallucination. Page numbers were not available in our copy. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

  • Within NLP, the definition of hallucination as generated content that is nonsensical or unfaithful to the provided source content is the most inclusive and standard.

    “Within the context of NLP, the above definition of hallucination, the generated content that is nonsensical or unfaithful to the provided source content [72, 177, 198, 325], is the most inclusive and standard.”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [198]

    The Introduction defines hallucination as generated text that is fluent but unfaithful to its source, and the Hallucination section gives examples of plausible but unfaithful output. These passages support the unfaithfulness part of the citing sentence, but do not describe nonsensical output as hallucination or establish that this definition is the most inclusive and standard in NLP. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Hallucination problem still occurs even when there is very little divergence in dataset.

    “However, Parikh et al. [198] show that hallucination problem still occurs even when there is very little divergence in dataset.”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [198]

  • As a result, the target reference may contain information that cannot be supported by the source.

    “As a result, the target reference may contain information that cannot be supported by the source [198, 268].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [198]

  • Tolerance towards hallucinations is very low in both summarization and data-to-text tasks because it is essential to provide faithful generation.

    “Tolerance towards hallucinations is very low in both the summarization [197] and data-to-text tasks [198, 269, 274] because it is essential to provide faithful generation.”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [198]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 50 · Cited by 252

Research works printed but not linked: 40

  1. Sumit Asthana and Aaron Halfaker. 2018. With few eyes, all hoaxes are deep. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):21.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. In Proc. of EMNLP.
  3. Eva Banik, Claire Gardent, and Eric Kow. 2013. The kbgen challenge. In Proc. of European Workshop on NLG.
  4. David L Chen and Raymond J Mooney. 2008. Learn- ing to sportscast: a test of grounded language acqui- sition. In Proc. of ICML.
  5. Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020. Logical natural lan- guage generation from open-domain tables. In Proc. of ACL.
  6. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017a. Creating train- ing corpora for NLG micro-planning. In Proc. of ACL.
  7. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017b. The WebNLG challenge: Generating text from RDF data. In Proc. of INLG.
  8. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natu- ral language inference data. In Proc. of NAACL.
  9. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR.
  10. Philipp Koehn and Rebecca Knowles. 2017. Six chal- lenges for neural machine translation. In Proc. of WMT.
  11. Flip Korn, Xuezhi Wang, You Wu, and Cong Yu. 2019. Automatically generating interesting facts from wikipedia tables. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD 19, page 349361, New York, NY, USA. Association for Computing Machinery. · DOI
  12. Wojciech Kryściński, Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proc. of EMNLP.
  13. Karen Kukich. 1983. Design of a knowledge-based re- port generator. In Proc. of ACL.
  14. J. Richard Landis and Gary G. Koch. 1977. The mea- surement of observer agreement for categorical data. Biometrics, 33(1):159-174.
  15. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proc. of EMNLP.
  16. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. In Open Review.
  17. Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning semantic correspondences with less super- vision. In Proc. of ACL.
  18. Greg Little, Lydia B Chilton, Max Goldman, and Robert C Miller. 2010. Turkit: human computation algorithms on mechanical turk. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 57-66.
  19. Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning. In Proc. of AAAI.
  20. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong mt sys- tems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90.
  21. Inderjeet Mani. 1999. Advances in automatic text sum- marization. MIT press.
  22. Kathleen McKeown. 1992. Text generation. Cam- bridge University Press.
  23. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to- end generation. In Proc. of SIGDIAL.
  24. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing nlg data: Pictures elicit bet- ter data. In Proc. of INLG.
  25. Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. 2018. Exemplar encoder-decoder for neural conversation generation. In Proc. of ACL.
  26. Panupong Pasupat and Percy Liang. 2015. Composi- tional semantic parsing on semi-structured tables. In Proc. of ACL.
  27. Hao Peng, Ankur P Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. 2019. Text generation with exemplar-based adaptive decoding. In Proc. of NAACL.
  28. Laura Perez-Beltrachini and Mirella Lapata. 2018. Bootstrapping generators from noisy data. In Proc. of NAACL.
  29. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language infer- ence. In *SEM@NAACL-HLT.
  30. Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with content selection and planning. In Proc. of AAAI.
  31. Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Natural Lan- guage Engineering, 3(1):57-87.
  32. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for se- quence generation tasks. In Proc. of TACL.
  33. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proc. of ACL.
  34. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proc. of ACL.
  35. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proc. of NIPS.
  36. Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Con- fident decoding for faithful data-to-text generation. arXiv preprint arXiv:1910.08684. · arXiv
  37. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NIPS.
  38. Oriol Vinyals and Quoc Le. 2015. A neural conver- sational model. In Proc. of ICML Deep Learning Workshop.
  39. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei- Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural lan- guage generation for spoken dialogue systems. In Proc. of EMNLP.
  40. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document gen- eration. In Proc. of EMNLP.

Authors
Parikh, A., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D.
Affiliations
  • Google (United States) (US)
  • Georgia Institute of Technology (US)
ORCID
Venue
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Publisher
Association for Computational Linguistics
Published
2020
Pages
1173-1186
DOI
10.18653/v1/2020.emnlp-main.89
OpenAlex
W3022814719
Cited by
252
References
50
Citations by year
2026: 10 · 2025: 26 · 2024: 30 · 2023: 68 · 2022: 51 · 2021: 56 · 2020: 10 · 2019: 1
FWCI
17.65
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
table-to-text generation, controlled generation, datasets, text generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review