CoChat
Sign Up
PreprintCited by 239Topic ModelingMultimodal Machine Learning ApplicationsAdvanced Graph Neural Networks

The DOI matches its DataCite record.

Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling

Open access
Yang, Linyao, Chen, Hongyang, Li, Zhao, Ding, Xiao, Wu, Xindong·2023·arXiv
DOI 10.48550/arxiv.2306.11489arXiv 2306.11489
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, ChatGPT, a representative large language model (LLM), has gained considerable attention due to its powerful emergent abilities. Some researchers suggest that LLMs could potentially replace structured knowledge bases like knowledge graphs (KGs) and function as parameterized knowledge bases. However, while LLMs are proficient at learning probabilistic language patterns based on large corpus and engaging in conversations with humans, they, like previous smaller pre-trained language models (PLMs), still have difficulty in recalling facts while generating knowledge-grounded contents. To overcome these limitations, researchers have proposed enhancing data-driven PLMs with knowledge-based KGs to incorporate explicit factual knowledge into PLMs, thus improving their performance to generate texts requiring factual knowledge and providing more informed responses to user queries. This paper reviews the studies on enhancing PLMs with KGs, detailing existing knowledge graph enhanced pre-trained language models (KGPLMs) as well as their applications. Inspired by existing studies on KGPLM, this paper proposes to enhance LLMs with KGs by developing knowledge graph-enhanced large language models (KGLLMs). KGLLM provides a solution to enhance LLMs' factual reasoning ability, opening up new avenues for LLM research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.11489

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Gaining access to external tools endows LLMs with real-time factual knowledge, multimodal functionalities, and specialized skills in vertical domains.

    “Gaining access to external tools endows LLMs with real-time factual knowledge (Yang et al., 2023), multimodal functionalities (Gupta & Kembhavi, 2023), and specialized skills in vertical domains (Jin et al., 2023).”

    From Counterfactually Auditable Lifecycle Certification for Autonomous Agents (2023) · cites this paper as Yang et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 150

  1. X. Zhou, C. Chai, G. Li, and J. Sun, "Database meets artificial intelligence: A survey," IEEE Trans. Knowl. Data Eng., vol. 34, no. 3, pp. 1096-1116, 2022.
  2. Q. Wang, Y. Li, R. Zhang, K. Shu, Z. Zhang, and A. Zhou, "A scalable query-aware enormous database generator for database evaluation," IEEE Trans. Knowl. Data Eng., vol. 35, no. 5, pp. 4395-4410, 2023.
  3. R. Lu, X. Jin, S. Zhang, M. Qiu, and X. Wu, "A study on big knowledge and its engineering issues," IEEE Trans. Knowl. Data Eng., vol. 31, no. 9, pp. 1630-1644, 2019.
  4. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," in Proc. of the 17th Annu. Conf. of the North Amer. Chapter of the Assoc. for Comput. Linguistics: Hum. Lang. Technol., 2019, pp. 4171-4186.
  5. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," J. of Mach. Learn. Res., vol. 21, no. 140, pp. 1-67, 2020.
  6. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler et al., "Emergent abilities of large language models," arXiv:2206.07682, 2022. · arXiv
  7. F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, "Language models as knowledge bases?" in Proc. 2019 Conf. Empirical Methods Nat. Lang. Process. and 9th Int. Joint Conf. Nat. Lang. Process., 2019, pp. 2463-2473.
  8. C. Wang, X. Liu, and D. Song, "Language models are open knowledge graphs," arXiv:2010.11967, 2020. · arXiv
  9. B. Heinzerling and K. Inui, "Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries," in Proc. 16th Conf. Eur. Chapter Assoc. Comput. Linguist., 2021, pp. 1772-1791.
  10. N. Bian, X. Han, L. Sun, H. Lin, Y. Lu, and B. He, "Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models," arXiv:2303.16421, 2023. · arXiv
  11. C. Wang, P. Liu, and Y. Zhang, "Can generative pre-trained language models serve as knowledge bases for closed-book qa?" in Proc. 59th Annu. Meet. Assoc. Comput. Linguist. and 11th Int. Joint Conf. Nat. Lang. Process., 2021, pp. 3241-3251.
  12. B. Cao, H. Lin, X. Han, L. Sun, L. Yan, M. Liao, T. Xue, and J. Xu, "Knowledgeable or educated guess? revisiting language models as knowledge bases," in Proc. 59th Annu. Meet. Assoc. Comput. Linguist. and 11th Int. Joint Conf. Nat. Lang. Process., 2021, pp. 1860-1874.
  13. H. Liu, R. Ning, Z. Teng, J. Liu, Q. Zhou, and Y. Zhang, "Evaluating the logical reasoning ability of chatgpt and gpt-4," arXiv:2304.03439, 2023. · arXiv
  14. Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung et al., "A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity," arXiv:2302.04023, 2023. · arXiv
  15. H. M. Yohannes and T. Amagasa, "Named-entity recognition for a low- resource language using pre-trained language model," in Proc. 37th ACM/SIGAPP Symp. Appl. Comput., 2022, p. 837-844.
  16. X. Wei, S. Wang, D. Zhang, P. Bhatia, and A. Arnold, "Knowledge enhanced pretrained language models: A compreshensive survey," arXiv:2110.08455, 2021. · arXiv
  17. L. Hu, Z. Liu, Z. Zhao, L. Hou, L. Nie, and J. Li, "A survey of knowledge enhanced pre-trained language models," IEEE Trans. Knowl. Data Eng., pp. 1-19, 2023.
  18. C. Zhen, Y. Shang, X. Liu, Y. Li, Y. Chen, and D. Zhang, "A survey on knowledge-enhanced pre-trained language models," arXiv:2212.13428, 2022. · arXiv
  19. X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang, "Self-supervised learning: Generative or contrastive," IEEE Trans. on Knowl. Data Eng., vol. 35, no. 1, pp. 857-876, 2023.
  20. K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning, "ELECTRA: Pre-training text encoders as discriminators rather than generators," in Proc. 8th Int. Conf. Learn. Representations, 2020.
  21. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, and O. Levy, "SpanBERT: Improving pre-training by representing and predicting spans," Trans. Assoc. Comput. Linguist., vol. 8, pp. 64-77, 2020.
  22. Y. Wang, C. Sun, Y. Wu, J. Yan, P. Gao, and G. Xie, "Pre-training entity relation encoder with intra-span and inter-span information," in Proc. 2020 Conf. Empirical Methods Nat. Lang. Process., 2020, pp. 1692-1705.
  23. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, "Albert: A lite bert for self-supervised learning of language represen- tations," arXiv:1909.11942, 2019. · arXiv
  24. M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, "BART: Denoising sequence-to- sequence pre-training for natural language generation, translation, and comprehension," in Proc. 58th Ann. Meet. Assoc. Comput. Linguistics., 2020, pp. 7871-7880.
  25. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, "Deep contextualized word representations," arXiv:1802.05365, 2018. · arXiv
  26. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Proc. 31st Conf. Neural Inform. Process. Syst., 2017.
  27. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, "Roberta: A robustly optimized bert pretraining approach," arXiv:1907.11692, 2019. · arXiv
  28. V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter," arXiv:1910.01108, 2019. · arXiv
  29. Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, and Q. Liu, "ERNIE: Enhanced language representation with informative entities," in Proc. 57th Ann. Meet. Assoc. Comput. Linguistics., 2019, pp. 1441-1451.
  30. P. He, X. Liu, J. Gao, and W. Chen, "Deberta: Decoding-enhanced bert with disentangled attention," in International Conference on Learning Representations, 2021.
  31. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, pp. 1-9, 2019.
  32. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, "Xlnet: Generalized autoregressive pretraining for language understanding," in Advances in neural information processing systems, 2019.
  33. T. Shen, Y. Mao, P. He, G. Long, A. Trischler, and W. Chen, "Ex- ploiting structured knowledge in text via graph-guided representation learning," in Proc. 2020 Conf. Empirical Methods Nat. Lang. Process., 2020, p. 8980-8994.
  34. W. Fedus, B. Zoph, and N. Shazeer, "Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity," The Journal of Machine Learning Research, vol. 23, no. 1, pp. 5232-5270, 2022.
  35. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma et al., "Scaling instruction-finetuned language models," arXiv:2210.11416, 2022. · arXiv
  36. A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia et al., "Glm-130b: An open bilingual pre-trained model," arXiv:2210.02414, 2022. · arXiv
  37. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, and A. Askell, "Language models are few-shot learners," in Adv. Neural Inform. Process. Syst., 2020, pp. 1877-1901.
  38. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, B. Zoph, L. Fedus, M. P. Bosma, Z. Zhou, T. Wang, E. Wang, K. Webster, M. Pellat, K. Robinson, K. Meier- Hellstern, T. Duke, L. Dixon, K. Zhang, Q. Le, Y. Wu, Z. Chen, and C. Cui, "GLaM: Efficient scaling of language models with mixture-of- experts," in Proc. 39th Int. Conf. Machine Learning, 2022, pp. 5547- 5569.
  39. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, and A. Ray, "Training language models to follow instructions with human feedback," in Adv. Neural Inform. Process. Syst., 2022, pp. 27 730-27 744.
  40. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann et al., "Palm: Scaling language modeling with pathways," arXiv:2204.02311, 2022. · arXiv
  41. R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.- T. Cheng, A. Jin, T. Bos, L. Baker, and Y. Du, "Lamda: Language models for dialog applications," arXiv:2201.08239, 2022. · arXiv
  42. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin et al., "Opt: Open pre-trained transformer language models," arXiv:2205.01068, 2022. · arXiv
  43. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., "Llama: Open and efficient foundation language models," arXiv:2302.13971, 2023. · arXiv
  44. OpenAI, "Gpt-4 technical report," arXiv:2303.08774, 2023. · arXiv
  45. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv:2307.09288, 2023. · arXiv
  46. W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong et al., "A survey of large language models," arXiv:2303.18223, 2023. · arXiv
  47. J.-W. Lu, C. Guo, X.-Y. Dai, Q.-H. Miao, X.-X. Wang, J. Yang, and F.-Y. Wang, "The chatgpt after: Opportunities and challenges of very large scale pre-trained models," Acta Autom. Sin., vol. 49, no. 4, pp. 705-717, 2023.
  48. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learning from human preferences," in Adv. Neural Inf. Process. Syst., 2017.
  49. P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, "Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing," ACM Comput. Surv., vol. 55, no. 9, pp. 1-35, 2023.
  50. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv:1707.06347, 2017. · arXiv
  51. J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou, "Chain of thought prompting elicits reasoning in large language models," arXiv:2201.11903, 2022. · arXiv
  52. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Ka- mar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al., "Sparks of artificial general intelligence: Early experiments with gpt-4," arXiv:2303.12712, 2023. · arXiv
  53. Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, "A survey for in-context learning," arXiv:2301.00234, 2022. · arXiv
  54. J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, "Finetuned language models are zero-shot learners," arXiv:2109.01652, 2021. · arXiv
  55. Y. Qin, S. Hu, Y. Lin, W. Chen, N. Ding, G. Cui, Z. Zeng, Y. Huang, C. Xiao, and C. Han, "Tool learning with foundation models," arXiv:2304.08354, 2023. · arXiv
  56. K. Yang, Y. Tian, N. Peng, and K. Dan, "Re3: Generating longer stories with recursive reprompting and revision," in Proc. 2022 Conf. Empir. Methods Nat. Lang. Process., 2022, p. 4393-4479.
  57. W. Yu, C. Zhu, Z. Li, Z. Hu, Q. Wang, H. Ji, and M. Jiang, "A survey of knowledge-enhanced text generation," ACM Comput. Surv., vol. 54, no. 11, pp. 1-18, 2022.
  58. X. Li, R. Zhao, Y. K. Chia, B. Ding, L. Bing, S. Joty, and S. Poria, "Chain of knowledge: A framework for grounding large language models with structured knowledge bases," arXiv:2305.13269, 2023. · arXiv
  59. E. Yanai, K. Nora, R. Shauli, F. Amir, R. Abhilasha, M. Marius, B. Yonatan, S. Hinrich, and G. Yoav, "Measuring causal effects of data statistics on language model's 'factual' predictions," arXiv:2207.14251, 2022. · arXiv
  60. J. Sun, C. Xu, L. Tang, S. Wang, C. Lin, Y. Gong, H.-Y. Shum, and J. Guo, "Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph," arXiv:2307.07697, 2023. · arXiv
  61. S. Abulhair and H. He, "Language models are greedy reasoners: A systematic formal analysis of chain-of-thought," arXiv:2210.01240, 2022. · arXiv
  62. T. A. Chang and B. K. Bergen, "Language model behavior: A com- prehensive survey," arXiv:2303.11504, 2023. · arXiv
  63. S. Wang, Z. Wei, J. Xu, and Z. Fan, "Unifying structure rea- soning and language model pre-training for complex reasoning," arXiv:2301.08913, 2023. · arXiv
  64. X. Zhang, A. Bosselut, M. Yasunaga, H. Ren, P. Liang, C. D. Manning, and J. Leskovec, "Greaselm: Graph reasoning enhanced language models for question answering," arXiv:2201.08860, 2022. · arXiv
  65. R. Ding, X. Han, and L. Wang, "A unified knowledge graph augmen- tation service for boosting domain-specific nlp tasks," in Find. Assoc. Comput. Linguist.: ACL 2023, 2023, pp. 353-369.
  66. J. Baek, A. F. Aji, and A. Saffari, "Knowledge-augmented language model prompting for zero-shot knowledge graph question answering," in Proc. 1st Workshop Nat. Lang. Reasoning Struct. Expl., 2023, pp. 78-106.
  67. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, "Re- alToxicityPrompts: Evaluating neural toxic degeneration in language models," in Find. Assoc. Comput. Linguist.: EMNLP 2020, 2020, pp. 3356-3369.
  68. R. Zellers, A. Holtzman, E. Clark, L. Qin, A. Farhadi, and Y. Choi, "TuringAdvice: A generative and dynamic evaluation of language use," in Proc. 2021 Conf. North Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol., 2021, pp. 4856-4880.
  69. V. Swamy, A. Romanou, and M. Jaggi, "Interpreting language models through knowledge graph extraction," arXiv:2111.08546, 2021. · arXiv
  70. W. Liu, P. Zhou, Z. Zhao, Z. Wang, Q. Ju, H. Deng, and P. Wang, "K-bert: Enabling language representation with knowledge graph," in Proc. AAAI Conf. Artif. Intell., 2020, pp. 2901-2908.
  71. T. Sun, Y. Shao, X. Qiu, Q. Guo, Y. Hu, X. Huang, and Z. Zhang, "CoLAKE: Contextualized language and knowledge embedding," in Proc. 28th Int. Conf. Comput. Linguistics, 2020, pp. 3660-3670.
  72. Y. Zhang, J. Lin, Y. Fan, P. Jin, Y. Liu, and B. Liu, "Cn-hit-it. nlp at semeval-2020 task 4: Enhanced language representation with multiple knowledge triples," in Proc. 14th Workshop Semant. Eval., 2020, pp. 494-500.
  73. I. Yamada, A. Asai, H. Shindo, H. Takeda, and Y. Matsumoto, "LUKE: Deep contextualized entity representations with entity-aware self- attention," in Proc. 2020 Conf. Empir. Methods Nat. Lang. Process., 2020, pp. 6442-6454.
  74. N. Poerner, U. Waltinger, and H. Schütze, "E-BERT: Efficient-yet- effective entity embeddings for BERT," in Find. Assoc. Comput. Linguist.: EMNLP 2020, 2020, pp. 803-818.
  75. C. Rosset, C. Xiong, M. Phan, X. Song, P. Bennett, and S. Tiwary, "Knowledge-aware language model pretraining," arXiv:2007.00655, 2020. · arXiv
  76. X. Liu, D. Yin, J. Zheng, X. Zhang, P. Zhang, H. Yang, Y. Dong, and J. Tang, "Oag-bert: Towards a unified backbone language model for academic knowledge services," in Proc. 28th ACM SIGKDD Conf. Knowl. Discov. Data Min., 2022, p. 3418-3428.
  77. T. Zhang, C. Wang, N. Hu, M. Qiu, C. Tang, X. He, and J. Huang, "Dkplm: Decomposable knowledge-enhanced pre-trained language model for natural language understanding," in Proc. AAAI Conf. Artif. Intell., 2022, pp. 11 703-11 711.
  78. Z.-X. Ye, Q. Chen, W. Wang, and Z.-H. Ling, "Align, mask and select: A simple method for incorporating commonsense knowledge into language representation models," arXiv:1908.06725, 2019. · arXiv
  79. W. Chen, Y. Su, X. Yan, and W. Y. Wang, "KGPT: Knowledge- grounded pre-training for data-to-text generation," in Proc. 2020 Conf. Empir. Methods Nat. Lang. Process., 2020, pp. 8635-8648.
  80. R. Logan, N. F. Liu, M. E. Peters, M. Gardner, and S. Singh, "Barack's wife hillary: Using knowledge graphs for fact-aware language model- ing," in Proc. 57th Annu. Meet. Assoc. Comput. Linguist., 2019, pp. 5962-5971.
  81. M. Sap, R. Le Bras, E. Allaway, C. Bhagavatula, N. Lourie, H. Rashkin, B. Roof, N. A. Smith, and Y. Choi, "Atomic: An atlas of machine commonsense for if-then reasoning," in Proc. 33rd AAAI Conf. Artif. Intell. & 31st Innov. Appl. Artif. Intell. Conf. & 9th AAAI Symp. Educ. Adv. Artif. Intell., 2019, p. 3027-3035.
  82. X. Wang, T. Gao, Z. Zhu, Z. Zhang, Z. Liu, J. Li, and J. Tang, "Kepler: A unified model for knowledge embedding and pre-trained language representation," Trans. Assoc. Comput. Linguist., vol. 9, pp. 176-194, 2021.
  83. W. Xiong, J. Du, W. Y. Wang, and V. Stoyanov, "Pretrained ency- clopedia: Weakly supervised knowledge-pretrained language model," arXiv:1912.09637, 2019. · arXiv
  84. Y. Sun, S. Wang, S. Feng, S. Ding, C. Pang, J. Shang, J. Liu, X. Chen, Y. Zhao, Y. Lu et al., "Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation," arXiv:2107.02137, 2021. · arXiv
  85. B. He, D. Zhou, J. Xiao, X. Jiang, Q. Liu, N. J. Yuan, and T. Xu, "BERT-MK: Integrating graph contextualized knowledge into pre- trained language models," in Find. Assoc. Comput. Linguist.: EMNLP 2020, 2020, pp. 2281-2290.
  86. Y. Su, X. Han, Z. Zhang, Y. Lin, P. Li, Z. Liu, J. Zhou, and M. Sun, "Cokebert: Contextual knowledge selection and embedding towards enhanced pre-trained language models," AI Open, vol. 2, pp. 127-134, 2021.
  87. Y. Sun, Q. Shi, L. Qi, and Y. Zhang, "JointLK: Joint reasoning with language models and knowledge graphs for commonsense question answering," in Proc. 2022 Conf. North Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol., 2022, pp. 5049-5060.
  88. P. Zhong, D. Wang, and C. Miao, "Knowledge-enriched transformer for emotion detection in textual conversations," in Proc. 2019 Conf. Empir. Methods Nat. Lang. Process. & 9th Int. Joint Conf. Nat. Lang. Process., 2019, pp. 165-176.
  89. Q. Liu, D. Yogatama, and P. Blunsom, "Relational memory-augmented language models," Trans. Assoc. Comput. Linguist., vol. 10, pp. 555- 572, 2022.
  90. M. Yasunaga, H. Ren, A. Bosselut, P. Liang, and J. Leskovec, "QA- GNN: Reasoning with language models and knowledge graphs for question answering," in Proc. 2021 Conf. North Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol., Online, 2021, pp. 535-546.
  91. L. He, S. Zheng, T. Yang, and F. Zhang, "KLMo: Knowledge graph enhanced pretrained language model with fine-grained relationships," in Find. Assoc. Comput. Linguist.: EMNLP 2021, 2021, pp. 4536-4542.
  92. M. E. Peters, M. Neumann, R. Logan, R. Schwartz, V. Joshi, S. Singh, and N. A. Smith, "Knowledge enhanced contextual word representa- tions," in Proc. 2019 Conf. Empir. Methods Nat. Lang. Process. & 9th Int. Joint Conf. Nat. Lang. Process., 2019, pp. 43-54.
  93. D. Yu, C. Zhu, Y. Yang, and M. Zeng, "Jaket: Joint pre-training of knowledge graph and language understanding," in Proc. AAAI Conf. Artif. Intell., 2022, pp. 11 630-11 638.
  94. Y. Liu, Y. Wan, L. He, H. Peng, and S. Y. Philip, "Kg-bart: Knowledge graph-augmented bart for generative commonsense reasoning," in Proc. AAAI Conf. Artif. Intell., 2021, pp. 6418-6425.
  95. R. Wang, D. Tang, N. Duan, Z. Wei, X. Huang, G. Cao, D. Jiang, and M. Zhou, "K-adapter: Infusing knowledge into pre-trained models with adapters," in Proc. Joint Conf. 59th Annu. Meet. Assoc. Comput. Linguist. and 11th Int. Joint Conf. Nat. Lang. Process., 2021, p. 1405-1418.
  96. A. Lauscher, O. Majewska, L. F. R. Ribeiro, I. Gurevych, N. Rozanov, and G. Glavaš, "Common sense or world knowledge? investigating adapter-based knowledge injection into pretrained transformers," in Proc. DeeLIO: 1st Workshop Knowl. Extract. Integr. Deep Learn. Archit., 2020, pp. 43-49.
  97. Q. Lu, D. Dou, and T. H. Nguyen, "Parameter-efficient domain knowledge integration from multiple sources for biomedical pre-trained language models," in Find. Assoc. Comput. Linguist.: EMNLP 2021, 2021, pp. 3855-3865.
  98. G. Lu, H. Yu, Z. Yan, and Y. Xue, "Commonsense knowledge graph- based adapter for aspect-level sentiment classification," Neurocomput., vol. 534, pp. 67-76, 2023.
  99. Y. Levine, B. Lenz, O. Dagan, O. Ram, D. Padnos, O. Sharir, S. Shalev- Shwartz, A. Shashua, and Y. Shoham, "SenseBERT: Driving some sense into BERT," in Proc. 58th Annu. Meet. Assoc. Comput. Linguist., 2020, pp. 4656-4667.
  100. Y. Qin, Y. Lin, R. Takanobu, Z. Liu, P. Li, H. Ji, M. Huang, M. Sun, and J. Zhou, "ERICA: Improving entity and relation understanding for pre-trained language models via contrastive learning," in Proc. 59th Ann. Meet. Assoc. Comput. Ling. Int. Jt. Conf. Nat. Lang. Process., 2021, pp. 3350-3363.
  101. P. Ke, H. Ji, S. Liu, X. Zhu, and M. Huang, "SentiLARE: Sentiment- aware language representation learning with linguistic knowledge," in Proc. 2020 Conf. Empir. Methods Nat. Lang. Process., Online, 2020, pp. 6975-6988.
  102. Y. Sun, S. Wang, Y. Li, S. Feng, H. Tian, H. Wu, and H. Wang, "Ernie 2.0: A continual pre-training framework for language understanding," in Proc. AAAI Conf. Artif. Intell., 2020, pp. 8968-8975.
  103. M. Yasunaga, A. Bosselut, H. Ren, X. Zhang, C. D. Manning, P. S. Liang, and J. Leskovec, "Deep bidirectional language-knowledge graph pretraining," in Adv. Neural Inform. Process. Syst., 2022, pp. 37 309- 37 323.
  104. H. Hayashi, Z. Hu, C. Xiong, and G. Neubig, "Latent relation language models," in Proc. AAAI Conf. Artif. Intell., 2020, pp. 7911-7918.
  105. M. Kang, J. Baek, and S. J. Hwang, "KALA: knowledge-augmented language model adaptation," in Proc. 2022 Conf. North Am. Chapter Assoc. Comput. Linguist.: Hum. Lang. Technol., 2022, pp. 5144-5167.
  106. Q. Xie, J. A. Bishop, P. Tiwari, and S. Ananiadou, "Pre-trained language models with domain knowledge for biomedical extractive summarization," Knowl. Based Syst., vol. 252, p. 109460, 2022.
  107. B. Y. Lin, X. Chen, J. Chen, and X. Ren, "KagNet: Knowledge-aware graph networks for commonsense reasoning," in Proc. 2019 Conf. Empir. Methods Nat. Lang. Process. & 9th Int. Joint Conf. Nat. Lang. Process., 2019, pp. 2829-2839.
  108. H. Fei, Y. Ren, Y. Zhang, D. Ji, and X. Liang, "Enriching contextualized language model from knowledge graph for biomedical information extraction," Brief. Bioinform., vol. 22, no. 3, p. bbaa110, 2021.
  109. T.-Y. Chang, Y. Liu, K. Gopalakrishnan, B. Hedayatnia, P. Zhou, and D. Hakkani-Tur, "Incorporating commonsense knowledge graph in pretrained models for social commonsense tasks," in Proc. DeeLIO: 1st Workshop Knowl. Extract. Integr. Deep Learn. Archit., Nov. 2020, pp. 74-79.
  110. B. R. Andrus, Y. Nasiri, S. Cui, B. Cullen, and N. Fulda, "Enhanced story comprehension for large language models through dynamic document-based knowledge graphs," in Proc. AAAI Conf. Artif. Intell., 2022, pp. 10 436-10 444.
  111. J. Wang, W. Huang, Q. Shi, H. Wang, M. Qiu, X. Li, and M. Gao, "Knowledge prompting in pre-trained language model for natural language understanding," in Proc. 2022 Conf. Empir. Methods Nat. Lang. Process., 2022, pp. 3164-3177.
  112. A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko, "Translating embeddings for modeling multi-relational data," in Adv. Neural Inf. Process. Syst., 2013.
  113. A. Wettig, T. Gao, Z. Zhong, and D. Chen, "Should you mask 15% in masked language modeling?" arXiv:2202.08005, 2022. · arXiv
  114. Z. Bi, N. Zhang, Y. Jiang, S. Deng, G. Zheng, and H. Chen, "When do program-of-thoughts work for reasoning?" arXiv:2308.15452, 2023. · arXiv
  115. T. Schick and H. Schütze, "Exploiting cloze-questions for few-shot text classification and natural language inference," in Proc. 16th Conf. Eur. Chapter Assoc. Comput. Linguist., 2021, pp. 255-269.
  116. N. Bian, X. Han, B. Chen, and L. Sun, "Benchmarking knowledge- enhanced commonsense question answering via knowledge-to-text transformation," in Proc. AAAI Conf. Artif. Intell., 2021, pp. 12 574- 12 582.
  117. J. Giorgi, X. Wang, N. Sahar, W. Y. Shin, G. D. Bader, and B. Wang, "End-to-end named entity recognition and relation extraction using pre- trained language models," arXiv:1912.13415, 2019. · arXiv
  118. Z. Yuan, Y. Liu, C. Tan, S. Huang, and F. Huang, "Improving biomed- ical pretrained language models with knowledge," in Proc. BioNLP 2021 workshop, 2021, pp. 180-190.
  119. D. Seyler, T. Dembelova, L. Del Corro, J. Hoffart, and G. Weikum, "A study of the importance of external knowledge in the named entity recognition task," in Proc. 56th Ann. Meet. Assoc. Comput. Linguistics., 2018, pp. 241-246.
  120. Q. He, L. Wu, Y. Yin, and H. Cai, "Knowledge-graph augmented word representations for named entity recognition," in Proc. AAAI Conf. Artif. Intell., 2020, pp. 7919-7926.
  121. Y. Song, W. Zhang, Y. Ye, C. Zhang, and K. Zhang, "Knowledge- enhanced relation extraction in chinese emrs," in Proc. 2022 5th Int. Conf. Mach. Learn. Nat. Lang. Process., 2023, p. 196-201.
  122. J. Li, Y. Katsis, T. Baldwin, H.-C. Kim, A. Bartko, J. McAuley, and C.-N. Hsu, "Spot: Knowledge-enhanced language representations for information extraction," in Proc. 31st ACM Int. Conf. Inf. Knowl. Manage., 2022, p. 1124-1134.
  123. A. Roy and S. Pan, "Incorporating medical knowledge in BERT for clinical relation extraction," in Proc. 2021 Conf. Empir. Methods Nat. Lang. Process., 2021, pp. 5357-5366.
  124. J. Zhou, J. Tian, R. Wang, Y. Wu, W. Xiao, and L. He, "SentiX: A sentiment-aware pre-trained model for cross-domain sentiment analy- sis," in Proc. 28th Int. Conf. Comput. Linguist., 2020, pp. 568-579.
  125. Q. Wang, X. Cao, J. Wang, and W. Zhang, "Knowledge-aware col- laborative filtering with pre-trained language model for personalized review-based rating prediction," IEEE Trans. Knowl. Data Eng., pp. 1-13, 2023.
  126. S. Liang, J. Shao, D. Zhang, J. Zhang, and B. Cui, "Drgi: Deep relational graph infomax for knowledge graph completion," IEEE Trans. Knowl. Data Eng., vol. 35, no. 3, pp. 2486-2499, 2023.
  127. Q. Lin, R. Mao, J. Liu, F. Xu, and E. Cambria, "Fusing topology contexts and logical rules in language models for knowledge graph completion," Inf. Fusion, vol. 90, pp. 253-264, 2023.
  128. W. Li, R. Peng, and Z. Li, "Knowledge graph completion by jointly learning structural features and soft logical rules," IEEE Trans. Knowl. Data Eng., vol. 35, no. 3, pp. 2724-2735, 2023.
  129. A. Ghanbarpour and H. Naderi, "An attribute-specific ranking method based on language models for keyword search over graphs," IEEE Trans. Knowl. Data Eng., vol. 32, no. 1, pp. 12-25, 2020.
  130. Z. Hu, Y. Xu, W. Yu, S. Wang, Z. Yang, C. Zhu, K.-W. Chang, and Y. Sun, "Empowering language models with knowledge graph reasoning for open-domain question answering," in Proc. 2022 Conf. Empir. Methods Nat. Lang. Process., 2022, pp. 9562-9581.
  131. J. Guan, F. Huang, Z. Zhao, X. Zhu, and M. Huang, "A knowledge- enhanced pretraining model for commonsense story generation," Trans. Assoc. Comput. Linguist., vol. 8, pp. 93-108, 2020.
  132. H. Ji, P. Ke, S. Huang, F. Wei, X. Zhu, and M. Huang, "Language generation with multi-hop reasoning on commonsense knowledge graph," in Proc. 2020 Conf. Empir. Methods Nat. Lang. Process., 2020, pp. 725-736.
  133. X. Yang and I. Tiddi, "Creative storytelling with language models and knowledge graphs," in Proc. CIKM 2020 Workshops, 2020.
  134. L. Du, X. Ding, T. Liu, and B. Qin, "Learning event graph knowledge for abductive reasoning," in Proc. 59th Annu. Meet. Assoc. Comput. Linguist. and 11th Int. Joint Conf. Nat. Lang. Process., 2021, pp. 5181- 5190.
  135. B. AlKhamissi, M. Li, A. Celikyilmaz, M. Diab, and M. Ghazvininejad, "A review on language models as knowledge bases," arXiv:2204.06031, 2022. · arXiv
  136. A. Talmor, Y. Elazar, Y. Goldberg, and J. Berant, "olmpics-on what language model pre-training captures," Trans. Assoc. Comput. Linguist., vol. 8, pp. 743-758, 2020.
  137. C. Wang, S. Liang, Y. Zhang, X. Li, and T. Gao, "Does it make sense? and why? a pilot study for sense making and explanation," in Proc. 57th Ann. Meet. Assoc. Comput. Linguistics., 2019, pp. 4020-4026.
  138. M. Sung, J. Lee, S. Yi, M. Jeon, S. Kim, and J. Kang, "Can language models be biomedical knowledge bases?" in Proc. 2021 Conf. Empir. Methods Nat. Lang. Process., 2021, pp. 4723-4734.
  139. R. Zhao, F. Zhao, G. Xu, S. Zhang, and H. Jin, "Can language models serve as temporal knowledge bases?" in Find. Assoc. Comput. Linguist.: EMNLP 2022, 2022, pp. 2024-2037.
  140. N. Kassner, P. Dufter, and H. Schütze, "Multilingual lama: In- vestigating knowledge in multilingual pretrained language models," arXiv:2102.00894, 2021. · arXiv
  141. T. Pires, E. Schlinger, and D. Garrette, "How multilingual is multilin- gual BERT?" in Proc. 57th Ann. Meet. Assoc. Comput. Linguistics., 2019, pp. 4996-5001.
  142. B. Cao, H. Lin, X. Han, F. Liu, and L. Sun, "Can prompt probe pretrained language models? understanding the invisible risks from a causal view," arXiv:2203.12258, 2022. · arXiv
  143. N. Kandpal, H. Deng, A. Roberts, E. Wallace, and C. Raffel, "Large language models struggle to learn long-tail knowledge," arXiv:2211.08411, 2022. · arXiv
  144. S. Hao, B. Tan, K. Tang, H. Zhang, E. P. Xing, and Z. Hu, "Bert- net: Harvesting knowledge graphs from pretrained language models," arXiv:2206.14268, 2022. · arXiv
  145. A. Bosselut, H. Rashkin, M. Sap, C. Malaviya, A. Celikyilmaz, and Y. Choi, "COMET: Commonsense transformers for automatic knowl- edge graph construction," in Proc. 57th Ann. Meet. Assoc. Comput. Linguistics., 2019, pp. 4762-4779.
  146. O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton- Brown, and Y. Shoham, "In-context retrieval-augmented language models," arXiv:2302.00083, 2023. · arXiv
  147. D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al., "Palm-e: An embodied multimodal language model," arXiv:2303.03378, 2023. · arXiv
  148. Y. Hou, G. Fu, and M. Sachan, "Understanding the integra- tion of knowledge in language models with graph convolutions," arXiv:2202.00964, 2022. · arXiv
  149. H. Schuff, H.-Y. Yang, H. Adel, and N. T. Vu, "Does external knowl- edge help explainable natural language inference? automatic evaluation vs. human ratings," in Proc. 4th BlackboxNLP Workshop on Analyz. Interpr. Neural Networks NLP, 2021, pp. 26-41.
  150. Z. Chen, A. K. Singh, and M. Sra, "Lmexplainer: a knowledge- enhanced explainer for language models," arXiv:2303.16537, 2023. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Yang, L., Chen, H., Li, Z., Ding, X., & Wu, X.
Affiliations
  • Zhejiang Lab (CN)
  • Harbin Institute of Technology (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
36
Issue
7
Pages
3091-3110
ISSN
1041-4347 (ISSN-L)
DOI
10.48550/arxiv.2306.11489
arXiv
2306.11489
OpenAlex
W4391407054
Cited by
239
References
0
Citations by year
2026: 70 · 2025: 111 · 2024: 55 · 2023: 3
FWCI
34.76
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Knowledge and Data Engineering (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (62306288, 62271452)
  • National Key Research and Development Program of China (2022PI0AC01, 2022YFB4500305)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
Keywords
large language models, knowledge graphs, factual reasoning, factual knowledge retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review