CoChat
Sign Up
Cited by 161Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications

Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training

Open access
Agarwal, Oshin, Ge, Heming, Shakeri, Siamak, Al-Rfou, Rami·2021·Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
DOI 10.18653/v1/2021.naacl-main.278
Open this paper in CoChat — free accountOpen PDF

Abstract

Oshin Agarwal, Heming Ge, Siamak Shakeri, Rami Al-Rfou. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.naacl-main.278

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Verbalization techniques convert structured data including knowledge graph triples, table rows, and database records into natural language sentences, enabling seamless integration with existing language systems without architectural modifications.

    “Verbalization techniques convert structured data including knowledge graph triples, table rows, and database records into natural language sentences, enabling seamless integration with existing language systems without architectural modifications [12, 788, 1072, 13].”

    From A Survey of Context Engineering for Large Language Models (2025) · cites this paper as [13]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. Oshin Agarwal, Mihir Kale, Heming Ge, Siamak Shak- eri, and Rami Al-Rfou. 2020. Machine transla- tion aided bilingual data-to-text generation and se- mantic parsing. In Proceedings of the 3rd Interna- tional Workshop on Natural Language Generation from the Semantic Web (WebNLG+), pages 125-130, Dublin, Ireland (Virtual). Association for Computa- tional Linguistics.
  2. Gabor Angeli, Melvin Jose Johnson Premkumar, and Christopher D. Manning. 2015. Leveraging linguis- tic structure for open domain information extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 344-354, Beijing, China. Association for Computa- tional Linguistics. · DOI · Paper page
  3. Jonathan Berant, Ido Dagan, and Jacob Goldberger. 2010. Global learning of focused entailment graphs. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1220-1229, Uppsala, Sweden. Association for Com- putational Linguistics.
  4. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Ad- vances in neural information processing systems, pages 4349-4357.
  5. Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In Proceedings of the 3rd International Work- shop on Natural Language Generation from the Se- mantic Web (WebNLG+), pages 55-76, Dublin, Ire- land (Virtual). Association for Computational Lin- guistics.
  6. Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019. Neu- ral data-to-text generation: A comparison between pipeline and end-to-end architectures. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 552-562, Hong Kong, China. Association for Computational Lin- guistics. · DOI
  7. Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020. Kgpt: Knowledge-grounded pre- training for data-to-text generation. arXiv preprint arXiv:2010.02307. · arXiv
  8. Ryan Clancy, Ihab F. Ilyas, and Jimmy Lin. 2019. Scal- able knowledge graph construction from text collec- tions. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pages 39-46, Hong Kong, China. Association for Compu- tational Linguistics. · DOI
  9. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language mod- els beyond a fixed-length context. arXiv preprint arXiv:1901.02860. · arXiv
  10. Rajarshi Das, Manzil Zaheer, Siva Reddy, and Andrew McCallum. 2017. Question answer- ing on knowledge bases and text using universal schema and memory networks. arXiv preprint arXiv:1704.08384. · arXiv
  11. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the e2e nlg challenge. arXiv preprint arXiv:1810.01170. · arXiv
  12. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh Interna- tional Conference on Language Resources and Eval- uation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  13. Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open information extrac- tion from the web. Communications of the ACM, 51(12):68-74.
  14. Thibault Févry, Livio Baldini Soares, Nicholas FitzGer- ald, Eunsol Choi, and Tom Kwiatkowski. 2020. En- tities as experts: Sparse memory access with entity supervision. arXiv preprint arXiv:2004.07202. · arXiv
  15. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The WebNLG challenge: Generating text from RDF data. In Pro- ceedings of the 10th International Conference on Natural Language Generation, pages 124-133, San- tiago de Compostela, Spain. Association for Compu- tational Linguistics. · DOI
  16. E. Goldberg, N. Driedger, and R. I. Kittredge. 1994. Using natural-language processing to produce weather forecasts. IEEE Expert, 9(2):45-53.
  17. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming-Wei Chang. 2020. Realm: Retrieval- augmented language model pre-training. arXiv preprint arXiv:2002.08909. · arXiv
  18. Annervaz K M, Somnath Basu Roy Chowdhury, and Ambedkar Dukkipati. 2018. Learning beyond datasets: Knowledge graph augmented neural net- works for natural language processing. In Proceed- ings of the 2018 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long Papers), pages 313-322, New Orleans, Louisiana. Association for Computational Linguis- tics. · DOI
  19. Mihir Kale. 2020. Text-to-text pre-training for data-to- text tasks. arXiv preprint arXiv:2005.10433. · arXiv
  20. Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text Generation from Knowledge Graphs with Graph Transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2284-2293, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
  21. Karen Kukich. 1983. Design of a knowledge-based re- port generator. In 21st Annual Meeting of the As- sociation for Computational Linguistics, pages 145- 150, Cambridge, Massachusetts, USA. Association for Computational Linguistics. · DOI
  22. Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceed- ings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203-1213, Austin, Texas. Association for Computational Lin- guistics. · DOI
  23. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401. · arXiv
  24. Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack's wife hillary: Using knowledge graphs for fact-aware lan- guage modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 5962-5971, Florence, Italy. Associa- tion for Computational Linguistics. · DOI · Paper page
  25. Thomas Manzini, Lim Yao Chong, Alan W Black, and Yulia Tsvetkov. 2019. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 615-621, Minneapo- lis, Minnesota. Association for Computational Lin- guistics. · DOI
  26. Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. Totto: A controlled table-to-text generation dataset. arXiv preprint arXiv:2004.14373. · arXiv
  27. Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge enhanced contextual word representations. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 43-54, Hong Kong, China. Associ- ation for Computational Linguistics. · DOI · Paper page
  28. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 2463-2473, Hong Kong, China. As- sociation for Computational Linguistics. · DOI · Paper page
  29. Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Nazneen Fatema Ra- jani, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, et al. 2020. Dart: Open-domain structured data record to text generation. arXiv preprint arXiv:2007.02871. · arXiv
  30. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the lim- its of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
  31. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics. · DOI · Paper page
  32. Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Pro- ceedings of the 2013 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74-84, Atlanta, Georgia. Association for Computa- tional Linguistics.
  33. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the param- eters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418-5426, Online. Association for Computational Linguistics.
  34. Anastasia Shimorina and Claire Gardent. 2018. Han- dling rare items in data-to-text generation. In Proceedings of the 11th International Conference on Natural Language Generation, pages 360-370, Tilburg University, The Netherlands. Association for Computational Linguistics. · DOI
  35. Robyn Speer and Catherine Havasi. 2012. Repre- senting general relational knowledge in Concept- Net 5. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3679-3686, Istanbul, Turkey. Eu- ropean Language Resources Association (ELRA).
  36. Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019. PullNet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 2380- 2390, Hong Kong, China. Association for Computa- tional Linguistics. · DOI
  37. Chris van der Lee, Emiel Krahmer, and Sander Wubben. 2018. Automated learning of templates for data-to-text generation: comparing rule-based, statistical and neural methods. In Proceedings of the 11th International Conference on Natural Lan- guage Generation, pages 35-45, Tilburg Univer- sity, The Netherlands. Association for Computa- tional Linguistics. · DOI
  38. Pat Verga, Haitian Sun, Livio Baldini Soares, and William W Cohen. 2020. Facts as experts: Adapt- able and interpretable neural memory over symbolic knowledge. arXiv preprint arXiv:2007.00849. · arXiv
  39. Denny Vrandečić and Markus Krötzsch. 2014. Wiki- data: A free collaborative knowledge base. Commu- nications of the ACM, 57:78-85.
  40. Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics. · DOI · Paper page
  41. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2018. Learning neural templates for text gen- eration. arXiv preprint arXiv:1808.10122. · arXiv
  42. Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Improving question answering over incomplete KBs with knowledge- aware reader. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 4258-4264, Florence, Italy. Associa- tion for Computational Linguistics. · DOI
  43. Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. 2020. Jaket: Joint pre-training of knowledge graph and language understanding. arXiv preprint arXiv:2010.00796. · arXiv

Cited by

Sorted by citations

Authors
Agarwal, O., Ge, H., Shakeri, S., & Al-Rfou, R.
Affiliations
  • University of Pennsylvania (US)
  • Google (United States) (US)
Venue
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Publisher
Association for Computational Linguistics
Published
2021
Pages
3554-3565
DOI
10.18653/v1/2021.naacl-main.278
OpenAlex
W3137014599
Cited by
161
References
7
Citations by year
2026: 12 · 2025: 23 · 2024: 34 · 2023: 45 · 2022: 36 · 2021: 11
FWCI
14.05
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
knowledge graphs, language model pretraining, synthetic data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review