CoChat
Sign Up
Cited by 524Topic ModelingMultimodal Machine Learning ApplicationsNatural Language Processing Techniques

Challenges in Data-to-Document Generation

Open access
Wiseman, Sam, Shieber, Stuart, Rush, Alexander·2017·Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/d17-1239
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent neural models have shown significant progress on the problem of generating short descriptive texts conditioned on a small number of database records.In this work, we suggest a slightly more difficult data-to-text generation task, and investigate how effective current approaches are on this task.In particular, we introduce a new, large-scale corpus of data records paired with descriptive documents, propose a series of extractive evaluation methods for analyzing performance, and obtain baseline results using current neural generation methods.Experiments show that these models produce fluent text, but fail to convincingly approximate humangenerated documents.Moreover, even templated baselines exceed the performance of these neural models on some metrics, though copy-and reconstructionbased extensions lead to noticeable improvements.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/d17-1239

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Some works heuristically select and pair real sentences or tables as source and target when collecting large-scale datasets.

    “When collecting large-scale datasets, some works heuristically select and pair real sentences or tables as the source and target [132, 283].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [283]

    The Introduction and §2 (Data-to-Text Datasets; page unavailable) describe a large-scale dataset pairing professionally written basketball-game summaries with corresponding statistical tables. They do not say that the authors heuristically selected the pairs, and the paired texts are documents rather than individual sentences. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Neural text generation models are prone to hallucinations because of the gap between structured data and text, which may cause semantic misunderstanding and erroneous correlation.

    “Although this field has been recently boosted by neural text generation models, it is well known that these models are prone to hallucinations [283] because of the gap between structured data and text, which may cause semantic misunderstanding and erroneous correlation.”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [283]

    Section 6.3 (page not provided) explicitly says a neural data-to-document model hallucinates factual statements unsupported by its source records and makes semantic errors. The paper does not establish the claimed causal chain—that a gap between structured data and text causes hallucinations, semantic misunderstanding, and erroneous correlation. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • There are general post-hoc IE-based metrics that could be applied to hallucination evaluation, such as Slot Error Rate, Content Selection, Relation Generation, and Content Ordering.

    “Moreover, there are some general posthoc IE-based metrics that could be applied to hallucination evaluation, such as Slot Error Rate (SER) [291], Content Selection (CS), Relation Generation (RG), and Content Ordering (CO) [268, 283].”

    From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [283]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. Gabor Angeli, Percy Liang, and Dan Klein. 2010. A simple domain-independent probabilistic approach to generation. In Proceedings of the 2010 Confer- ence on Empirical Methods in Natural Language Processing, pages 502-512. Association for Com- putational Linguistics.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
  3. Regina Barzilay and Mirella Lapata. 2005. Collective content selection for concept-to-text generation. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Lan- guage Processing, pages 331-338. Association for Computational Linguistics.
  4. Anja Belz. 2008. Automatic generation of weather forecast texts using comprehensive probabilistic generation-space models. Natural Language Engi- neering, 14(04):431-455.
  5. Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, An- drew M. Dai, Rafal Józefowicz, and Samy Ben- gio. 2016. Generating sentences from a continuous space. In CoNLL, pages 10-21.
  6. Eric Brill and Robert C Moore. 2000. An improved er- ror model for noisy channel spelling correction. In Proceedings of the 38th Annual Meeting on Associa- tion for Computational Linguistics, pages 286-293. Association for Computational Linguistics.
  7. David L Chen and Raymond J Mooney. 2008. Learn- ing to sportscast: a test of grounded language acqui- sition. In Proceedings of the 25th international con- ference on Machine learning, pages 128-135. ACM.
  8. Andrew Chisholm, Will Radford, and Ben Hachey. 2017. Learning to generate one-sentence biogra- phies from wikidata. CoRR, abs/1702.06235. · arXiv
  9. KyungHyun Cho, Bart van Merrienboer, Dzmitry Bah- danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder ap- proaches. Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation.
  10. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493-2537.
  11. Pablo A Duboue and Kathleen R McKeown. 2003. Sta- tistical acquisition of content selection rules for nat- ural language generation. In EMNLP, pages 121- 128. Association for Computational Linguistics.
  12. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179-211.
  13. Eli Goldberg, Norbert Driedger, and Richard I Kit- tredge. 1994. Using natural-language processing to produce weather forecasts. IEEE Expert, 9(2):45- 53.
  14. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In ACL.
  15. C ¸aglar Gülc ¸ehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In ACL.
  16. Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In ACL Volume 1: Long Papers.
  17. Dan Jurafsky and James H Martin. 2014. Speech and language processing, volume 3. Pearson London.
  18. Anjuli Kannan and Oriol Vinyals. 2016. Adversarial evaluation of dialogue models. In NIPS 2016 Work- shop on Adversarial Training.
  19. Joohyun Kim and Raymond J Mooney. 2010. Gen- erative alignment and semantic parsing for learn- ing from ambiguous supervision. In Proceedings of the 23rd International Conference on Computa- tional Linguistics: Posters, pages 543-551. Associ- ation for Computational Linguistics.
  20. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746- 1751.
  21. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Opennmt: Open-source toolkit for neural machine translation. CoRR, abs/1701.02810. · arXiv
  22. Karen Kukich. 1983. Design of a knowledge-based re- port generator. In ACL, pages 145-150.
  23. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In EMNLP, pages 1203-1213.
  24. Jiwei Li, Will Monroe, Tianlin Shi, Alan Ritter, and Dan Jurafsky. 2017. Adversarial learning for neural dialogue generation. CoRR, abs/1701.06547. · arXiv
  25. Percy Liang, Michael I Jordan, and Dan Klein. 2009. Learning semantic correspondences with less super- vision. In ACL, pages 91-99. Association for Com- putational Linguistics.
  26. Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation met- rics for dialogue response generation. In EMNLP, pages 2122-2132.
  27. Wei Lu and Hwee Tou Ng. 2011. A probabilistic forest-to-string model for language generation from typed lambda calculus expressions. In Proceedings of the Conference on Empirical Methods in Natu- ral Language Processing, pages 1611-1622. Asso- ciation for Computational Linguistics.
  28. Kathleen McKeown. 1992. Text generation -using dis- course strategies and focus constraints to generate natural language text. Studies in natural language processing. Cambridge University Press.
  29. Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2016. What to talk about and how? selective gener- ation using lstms with coarse-to-fine alignment. In NAACL HLT, pages 720-730.
  30. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. CoRR, abs/1609.07843. · arXiv
  31. Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Natural Lan- guage Engineering, 3(1):57-87.
  32. Ehud Reiter, Somayajulu Sripada, Jim Hunter, Jin Yu, and Ian Davy. 2005. Choosing words in computer- generated weather forecasts. Artificial Intelligence, 167(1-2):137-169.
  33. Jacques Robin. 1994. Revision-based generation of Natural Language Summaries providing historical Background. Ph.D. thesis, Citeseer.
  34. Cícero Nogueira dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. In ACL, pages 626- 634.
  35. Radu Soricut and Daniel Marcu. 2006. Stochastic lan- guage generation using widl-expressions and its ap- plication in machine translation and summarization. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Lin- guistics, pages 1105-1112. Association for Compu- tational Linguistics.
  36. Ilya Sutskever, James Martens, and Geoffrey E Hin- ton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML), pages 1017-1024.
  37. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural net- works. In Advances in Neural Information Process- ing Systems (NIPS), pages 3104-3112.
  38. Kumiko Tanaka-Ishii, Kôiti Hasida, and Itsuki Noda. 1998. Reactive content selection in the generation of real-time soccer commentary. In COLING-ACL, pages 1282-1288.
  39. Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In AAAI, pages 3097-3103.
  40. Michael White, Rajakrishnan Rajkumar, and Scott Martin. 2007. Towards broad coverage surface real- ization with ccg. In Proceedings of the Workshop on Using Corpora for NLG: Language Generation and Machine Translation (UCNLG+ MT), pages 267- 276.
  41. Yuk Wah Wong and Raymond J Mooney. 2007. Gen- eration by inverting a semantic parser that uses sta- tistical machine translation. In HLT-NAACL, pages 172-179.
  42. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural ma- chine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. · arXiv
  43. Zichao Yang, Phil Blunsom, Chris Dyer, and Wang Ling. 2016. Reference-aware language models. CoRR, abs/1611.01628. · arXiv
  44. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao, et al. 2014. Relation classification via convolutional deep neural network. In COLING, pages 2335-2344.
  45. Zhu Zhang. 2004. Weakly-supervised relation classifi- cation for information extraction. In Proceedings of the thirteenth ACM international conference on In- formation and knowledge management, pages 581- 588. ACM.
  46. GuoDong Zhou, JunHui Li, LongHua Qian, and Qiaoming Zhu. 2008. Semi-supervised learning for relation extraction. In Third International Joint Conference on Natural Language Processing, page 32.

Cited by

Sorted by citations

Authors
Wiseman, S., Shieber, S., & Rush, A.
Affiliations
Harvard University (US)
ORCID
Venue
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2017
Pages
2253-2263
DOI
10.18653/v1/d17-1239
OpenAlex
W2739046565
Cited by
524
References
3
Citations by year
2026: 10 · 2025: 28 · 2024: 22 · 2023: 73 · 2022: 62 · 2021: 110 · 2020: 94 · 2019: 87 · 2018: 38
FWCI
27.44
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
data-to-text generation, database records, corpus construction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review