CoChat
Sign Up
PreprintCited by 252Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

The DOI matches its DataCite record.

ToTTo: A Controlled Table-To-Text Generation Dataset

Open access
Parikh, Ankur P., Wang, Xuezhi, Gehrmann, Sebastian, Faruqui, Manaal, Dhingra, Bhuwan, Yang, Diyi, Das, Dipanjan·2020·arXiv
DOI 10.48550/arxiv.2004.14373arXiv 2004.14373
Open this paper in CoChat — free accountOpen PDF

Abstract

We present ToTTo, an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description. To obtain generated targets that are natural but also faithful to the source table, we introduce a dataset construction process where annotators directly revise existing candidate sentences from Wikipedia. We present systematic analyses of our dataset and annotation process as well as results achieved by several state-of-the-art baselines. While usually fluent, existing methods often hallucinate phrases that are not supported by the table, suggesting that this dataset can serve as a useful research benchmark for high-precision conditional text generation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2004.14373

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Each instruction in ToTTo is assigned one to three reference captions that were collected by authors of the original dataset.

    “Each instruction in ToTTo is assigned one to three reference captions that were collected by authors of the original dataset.”

    From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [59]

    Section 6.3 (page unavailable) says development and test examples have three annotator-produced final sentences used as references; Section 7 describes the target as a sentence generated from a table and highlighted cells. The paper does not establish that every “instruction” has one to three reference captions, and it describes the sentences as drawn from Wikipedia and revised by annotators rather than captions collected by the authors. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • In ToTTo, fullyspecified instructions have the following information elements: a HTML-formatted table extracted from a Wikipedia page, a subset of cells in the table that have been highlighted, the name of the Wikipedia page that included the Table, and the name of the Section in the Wikipedia page that included the Table.

    “In ToTTo, fullyspecified instructions have the following information elements: (1) a HTML-formatted table extracted from a Wikipedia page, (2) a subset of cells in the table that have been highlighted , (3) the name of the Wikipedia page that included the Table, (4) the name of the Section in the Wikipedia page that included the Table.”

    From LLMs Get Lost In Multi-Turn Conversation (2025) · cites this paper as [59]

    Section 7 (Machine Learning Task Construction) identifies a Wikipedia table, highlighted cells, page title, and section title as task inputs. Section 3 (Preliminaries) also lists section text as metadata, while our copy describes tables as structured cells rather than specifying that the input table is HTML-formatted. The four-item description is therefore supported in substance, but its format and completeness are not. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 48

  1. Sumit Asthana and Aaron Halfaker. 2018. With few eyes, all hoaxes are deep. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):21.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. In Proc. of EMNLP.
  3. Eva Banik, Claire Gardent, and Eric Kow. 2013. The kbgen challenge. In Proc. of European Workshop on NLG.
  4. David L Chen and Raymond J Mooney. 2008. Learn- ing to sportscast: a test of grounded language acqui- sition. In Proc. of ICML.
  5. Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020. Logical natural lan- guage generation from open-domain tables. In Proc. of ACL.
  6. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. TabFact: A large-scale dataset for table-based fact verification. In Proc. of ICLR.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proc. of NAACL.
  8. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William W Co- hen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In Proc. of ACL.
  9. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proc. of NAACL.
  10. Joseph L. Fleiss. 1971. Measuring nominal scale agree- ment among many raters. Psychological Bulletin, 76(5):378. · DOI · Paper page
  11. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017a. Creating train- ing corpora for NLG micro-planning. In Proc. of ACL.
  12. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017b. The WebNLG challenge: Generating text from RDF data. In Proc. of INLG.
  13. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natu- ral language inference data. In Proc. of NAACL.
  14. Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. TACL, 6:437-450.
  15. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR.
  16. Philipp Koehn and Rebecca Knowles. 2017. Six chal- lenges for neural machine translation. In Proc. of WMT.
  17. Flip Korn, Xuezhi Wang, You Wu, and Cong Yu. 2019. Automatically generating interesting facts from wikipedia tables. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD 19, page 349361, New York, NY, USA. Association for Computing Machinery. · DOI
  18. Wojciech Kryściński, Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proc. of EMNLP.
  19. Karen Kukich. 1983. Design of a knowledge-based re- port generator. In Proc. of ACL.
  20. J. Richard Landis and Gary G. Koch. 1977. The mea- surement of observer agreement for categorical data. Biometrics, 33(1):159-174.
  21. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proc. of EMNLP.
  22. Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2018. Hallucinations in neural machine translation. In Open Review.
  23. Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning semantic correspondences with less super- vision. In Proc. of ACL.
  24. Greg Little, Lydia B Chilton, Max Goldman, and Robert C Miller. 2010. Turkit: human computation algorithms on mechanical turk. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 57-66.
  25. Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning. In Proc. of AAAI.
  26. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong mt sys- tems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62-90.
  27. Inderjeet Mani. 1999. Advances in automatic text sum- marization. MIT press.
  28. Kathleen McKeown. 1992. Text generation. Cam- bridge University Press.
  29. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to- end generation. In Proc. of SIGDIAL.
  30. Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing nlg data: Pictures elicit bet- ter data. In Proc. of INLG.
  31. Gaurav Pandey, Danish Contractor, Vineet Kumar, and Sachindra Joshi. 2018. Exemplar encoder-decoder for neural conversation generation. In Proc. of ACL.
  32. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: a method for automatic eval- uation of machine translation. In Proc. of ACL.
  33. Panupong Pasupat and Percy Liang. 2015. Composi- tional semantic parsing on semi-structured tables. In Proc. of ACL.
  34. Hao Peng, Ankur P Parikh, Manaal Faruqui, Bhuwan Dhingra, and Dipanjan Das. 2019. Text generation with exemplar-based adaptive decoding. In Proc. of NAACL.
  35. Laura Perez-Beltrachini and Mirella Lapata. 2018. Bootstrapping generators from noisy data. In Proc. of NAACL.
  36. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language infer- ence. In *SEM@NAACL-HLT.
  37. Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with content selection and planning. In Proc. of AAAI.
  38. Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Natural Lan- guage Engineering, 3(1):57-87.
  39. Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for se- quence generation tasks. In Proc. of TACL.
  40. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proc. of ACL.
  41. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proc. of ACL.
  42. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proc. of NIPS.
  43. Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Con- fident decoding for faithful data-to-text generation. arXiv preprint arXiv:1910.08684. · arXiv
  44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NIPS.
  45. Oriol Vinyals and Quoc Le. 2015. A neural conver- sational model. In Proc. of ICML Deep Learning Workshop.
  46. Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei- Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural lan- guage generation for spoken dialogue systems. In Proc. of EMNLP.
  47. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document gen- eration. In Proc. of EMNLP.
  48. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proc. of ICCV.

Cited by

Sorted by citations

Metadata

Authors
Parikh, A. P., Wang, X., Gehrmann, S., Faruqui, M., Dhingra, B., Yang, D., & Das, D.
Affiliations
  • Google (United States) (US)
  • Georgia Institute of Technology (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
Pages
1173-1186
DOI
10.48550/arxiv.2004.14373
arXiv
2004.14373
OpenAlex
W3022814719
Cited by
252
References
0
Citations by year
2026: 10 · 2025: 26 · 2024: 30 · 2023: 68 · 2022: 51 · 2021: 56 · 2020: 10
FWCI
17.65
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
table-to-text generation, controlled generation, datasets, text generation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review