Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 6

The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.

Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications

Feng, Zhangyin, Ma, Weitao, Yu, Weijiang, Huang, Lei, Wang, Haotian, Chen, Qianglong, Peng, Weihua, Feng, Xiaocheng, Qin, Bing, liu, Ting·2023·arXiv
DOI 10.48550/arxiv.2311.05876arXiv 2311.05876
Open this paper in CoChat — free account

Abstract

Large language models (LLMs) exhibit superior performance on various natural language tasks, but they are susceptible to issues stemming from outdated data and domain-specific limitations. In order to address these challenges, researchers have pursued two primary strategies, knowledge editing and retrieval augmentation, to enhance LLMs by incorporating external information from different aspects. Nevertheless, there is still a notable absence of a comprehensive survey. In this paper, we propose a review to discuss the trends in integration of knowledge and large language models, including taxonomy of methods, benchmarks, and applications. In addition, we conduct an in-depth analysis of different methods and point out potential research directions in the future. We hope this survey offers the community quick access and a comprehensive overview of this research area, with the intention of inspiring future research endeavors.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2311.05876

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 3 references

Research works printed but not linked: 87

  1. Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopou- los, Oana Cocarascu, and Arpit Mittal. 2021. Fever- ous: Fact extraction and verification over unstruc- tured and structured information. arXiv preprint arXiv:2106.05707. · arXiv
  2. Berkeley R Andrus, Yeganeh Nasiri, Shilong Cui, Ben- jamin Cullen, and Nancy Fulda. 2022. Enhanced story comprehension for large language models through dynamic document-based knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10436-10444.
  3. Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023a. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136. · arXiv
  4. Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. 2022. Autoregressive search engines: Generating substrings as document identifiers. Advances in Neural Infor- mation Processing Systems, 35:31668-31683.
  5. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326. · arXiv
  6. Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing things from a different angle:discovering diverse perspec- tives about claims. In Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Pa- pers), pages 542-557, Minneapolis, Minnesota. As- sociation for Computational Linguistics. · DOI · Paper page
  7. Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. 2023. Measuring and improv- ing chain-of-thought reasoning in vision-language models. arXiv preprint arXiv:2309.04461. · arXiv
  8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. · arXiv
  9. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. In Machine learning challenges workshop, pages 177-190. Springer.
  10. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022a. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493- 8502.
  11. Damai Dai, Wenbin Jiang, Qingxiu Dong, Yajuan Lyu, Qiaoqiao She, and Zhifang Sui. 2022b. Neural knowledge bank for pretrained transformers. arXiv preprint arXiv:2208.00399. · arXiv
  12. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Edit- ing factual knowledge in language models. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 6491- 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
  13. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00904. · arXiv
  14. Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022. Calibrating factual knowledge in pretrained language models. In Find- ings of the Association for Computational Linguistics: EMNLP 2022, pages 5937-5947, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics. · DOI · Paper page
  15. Yihong Dong, Kangcheng Luo, Xue Jiang, Zhi Jin, and Ge Li. 2023. Pace: Improving prompt with actor-critic editing for large language model. arXiv preprint arXiv:2308.10088. · arXiv
  16. Yibing Du, Antoine Bosselut, and Christopher D Man- ning. 2022. Synthetic disinformation attacks on au- tomated fact verification systems. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 36, pages 10581-10589.
  17. Julian Eisenschlos, Bhuwan Dhingra, Jannis Bulian, Benjamin Börschinger, and Jordan Boyd-Graber. 2021. Fool me twice: Entailment from Wikipedia gamification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 352-365, Online. Association for Computational Linguistics. · DOI
  18. Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Bal- achandran, Tianxing He, and Yulia Tsvetkov. 2023a. Cook: Empowering general-purpose language mod- els with modular and collaborative knowledge. arXiv preprint arXiv:2305.09955. · arXiv
  19. Zhangyin Feng, Xiaocheng Feng, Dezhi Zhao, Maojin Yang, and Bing Qin. 2023c. Retrieval-generation synergy augmented large language models. arXiv preprint arXiv:2310.05149. · arXiv
  20. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023. RARR: Researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16477-16508, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  21. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021a. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346- 361.
  22. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021b. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
  23. Yuxuan Gu, Xiaocheng Feng, Sicheng Ma, Lingyuan Zhang, Heng Gong, and Bing Qin. 2022. A distri- butional lens for multi-aspect controllable text gen- eration. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1023-1043, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI
  24. Anshita Gupta, Debanjan Mondal, Akshay Krishna Sheshadri, Wenlong Zhao, Xiang Lorraine Li, Sarah Wiegreffe, and Niket Tandon. 2023. Edit- ing commonsense knowledge in GPT. CoRR, abs/2305.14956. · DOI
  25. Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309-2324, Online. Association for Computational Linguistics. · DOI
  26. Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2022. Aging with grace: Lifelong model editing with discrete key-value adaptors. arXiv preprint arXiv:2211.11031. · arXiv
  27. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghan- deharioun. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. CoRR, abs/2301.04213. · DOI · Paper page
  28. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 6609-6625, Barcelona, Spain (Online). Inter- national Committee on Computational Linguistics. · DOI · Paper page
  29. Linmei Hu, Zeyi Liu, Ziwang Zhao, Lei Hou, Liqiang Nie, and Juanzi Li. 2023. A survey of knowledge enhanced pre-trained language models. IEEE Trans- actions on Knowledge and Data Engineering.
  30. Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023b. Transformer- patcher: One mistake worth one neuron. arXiv preprint arXiv:2301.09785. · arXiv
  31. Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Worts- man, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2023. Editing models with task arithmetic. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  32. Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jan- nik Strötgen, and Gerhard Weikum. 2018. Tempques- tions: A benchmark for temporal question answering. In Companion Proceedings of the The Web Confer- ence 2018, pages 1057-1062.
  33. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Associa- tion for Computational Linguistics, 9:962-977. · DOI · Paper page
  34. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics. · DOI · Paper page
  35. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. · arXiv
  36. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In In- ternational Conference on Machine Learning, pages 15696-15707.
  37. PMLR.
  38. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6769-6781. Associa- tion for Computational Linguistics. · DOI · Paper page
  39. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics, 7:452-466. · DOI · Paper page
  40. Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. 2022. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115. · arXiv
  41. Hyunji Lee, Sohee Yang, Hanseok Oh, and Minjoon Seo. 2022. Generative multi-hop retrieval. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1417-1436.
  42. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. · arXiv
  43. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprint arXiv:2301.12597. · arXiv
  44. Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2023c. PMET: precise model editing in a transformer. CoRR, abs/2308.08742. · DOI
  45. Yongqi Li, Nan Yang, Liang Wang, Furu Wei, and Wen- jie Li. 2023d. Multiview identifiers enhanced gen- erative retrieval. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 6636-6648, Toronto, Canada. Association for Computational Lin- guistics. · DOI · Paper page
  46. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 7052-7063, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics. · DOI · Paper page
  47. Ziyang Luo, Can Xu, Pu Zhao, Xiubo Geng, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Augmented large language models with parametric knowledge guiding. arXiv preprint arXiv:2305.04757. · arXiv
  48. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. 2023. Query rewriting for retrieval- augmented large language models. arXiv preprint arXiv:2305.14283. · arXiv
  49. Aman Madaan, Niket Tandon, Peter Clark, and Yim- ing Yang. 2022. Memory-assisted prompt editing to improve GPT-3 after deployment. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2833-2861, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. · DOI · Paper page
  50. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric mem- ories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 9802-9822, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  51. Antonis Maronikolakis and Hinrich Schütze. 2021. Mul- tidomain pretrained language models for green NLP. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 1-8, Kyiv, Ukraine. Asso- ciation for Computational Linguistics.
  52. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associa- tions in GPT. In NeurIPS.
  53. Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. 2023. Mass- editing memory in a transformer. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  54. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022a. Fast model editing at scale. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  55. Eric Mitchell, Charles Lin, Antoine Bosselut, Christo- pher D Manning, and Chelsea Finn. 2022b. Memory- based model editing at scale. In International Con- ference on Machine Learning, pages 15817-15831.
  56. PMLR.
  57. Fedor Moiseev, Zhe Dong, Enrique Alfonseca, and Mar- tin Jaggi. 2022. SKILL: Structured knowledge infu- sion for large language models. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1581-1588, Seattle, United States. Association for Computational Linguistics. · DOI · Paper page
  58. Lin Padgham and Michael Winikoff. 2005. Developing intelligent agent systems: A practical guide. John Wiley & Sons.
  59. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics. · DOI · Paper page
  60. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models.
  61. Jing Qian, Li Dong, Yelong Shen, Furu Wei, and Weizhu Chen. 2022. Controllable natural language genera- tion with contrastive prefixes. In Findings of the As- sociation for Computational Linguistics: ACL 2022, pages 2912-2924, Dublin, Ireland. Association for Computational Linguistics. · DOI · Paper page
  62. Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends® in Information Re- trieval, 3(4):333-389.
  63. Amrita Saha, Vardaan Pahuja, Mitesh Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. Com- plex sequential question answering: Towards learn- ing to converse over linked question answer pairs with a knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
  64. Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. 2020. Commonsense reason- ing for natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 27-33, Online. Association for Computational Lin- guistics. · DOI
  65. Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Li- juan Wang. 2022. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09150. · arXiv
  66. Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitriy Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. arXiv preprint arXiv:2004.00345. · arXiv
  67. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics. · DOI
  68. Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bha- gavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2022. Commonsenseqa 2.0: Exposing the limits of ai through gamification. arXiv preprint arXiv:2201.05320. · arXiv
  69. Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. 2022. Transformer memory as a differentiable search index. Advances in Neural In- formation Processing Systems, 35:21831-21843.
  70. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
  71. Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, et al. 2022. A neural corpus indexer for document retrieval. Advances in Neural Information Processing Systems, 35:25600- 25614.
  72. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
  73. Suhang Wu, Minlong Peng, Yue Chen, Jinsong Su, and Mingming Sun. 2023. Eva-kellm: A new benchmark for evaluating knowledge editing of llms. CoRR, abs/2308.09954. · DOI
  74. Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, and Julian McAuley. 2023. Small models are valuable plug-ins for large language mod- els. arXiv preprint arXiv:2305.08848. · arXiv
  75. Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. 2023. Chatgpt is not enough: Enhanc- ing large language models with knowledge graphs for fact-aware language modeling. arXiv preprint arXiv:2306.11489. · arXiv
  76. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Com- putational Linguistics. · DOI · Paper page
  77. Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu, Xipeng Qiu, and Xuanjing Huang. 2023. Do large language models know what they don't know? In Findings of the Association for Computational Lin- guistics: ACL 2023, pages 8653-8665, Toronto, Canada. Association for Computational Linguistics. · DOI · Paper page
  78. Weijiang Yu, Jian Liang, Lei Ji, Lu Li, Yuejian Fang, Nong Xiao, and Nan Duan. 2021a. Hybrid reasoning network for video-based commonsense captioning. In Proceedings of the 29th ACM international con- ference on multimedia, pages 5213-5221.
  79. Weijiang Yu, Haofan Wang, Guohao Li, Nong Xiao, and Bernard Ghanem. 2023a. Knowledge-aware global reasoning for situation recognition. IEEE Transac- tions on Pattern Analysis and Machine Intelligence.
  80. Weijiang Yu, Haoteng Zheng, Mengfei Li, Lei Ji, Lijun Wu, Nong Xiao, and Nan Duan. 2021b. Learning from inside: Self-driven siamese sampling and rea- soning for video question answering. Advances in Neural Information Processing Systems, 34:26462- 26474.
  81. Wenhao Yu, Chenguang Zhu, Zaitang Li, Zhiting Hu, Qingyun Wang, Heng Ji, and Meng Jiang. 2022. A survey of knowledge-enhanced text generation. ACM Computing Surveys, 54(11s):1-38. · DOI · Paper page
  82. Zichun Yu, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. 2023d. Augmentation-adapted retriever improves generalization of language models as generic plug-in. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2421-2436, Toronto, Canada. Association for Computational Linguistics. · DOI
  83. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. · arXiv
  84. Michael Zhang and Eunsol Choi. 2021. SituatedQA: In- corporating extra-linguistic contexts into QA. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 7371- 7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI
  85. Zihan Zhang, Meng Fang, Ling Chen, Mohammad-Reza Namazi-Rad, and Jun Wang. 2023b. How do large language models capture the ever-changing world knowledge? a review of recent advances.
  86. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023b. A survey of large language models. Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305.12740. · arXiv
  87. Zexuan Zhong, Zhengxuan Wu, Christopher D Man- ning, Christopher Potts, and Danqi Chen. 2023. Mquake: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795. · arXiv

Cited by

See all 6

Metadata

Authors
Feng, Z., Ma, W., Yu, W., Huang, L., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., & liu, T.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2311.05876
arXiv
2311.05876
References
3
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications. https://app.cochat.ai/papers/10.48550/arxiv.2311.05876
The paper
Feng, Z., Ma, W., Yu, W., Huang, L., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., & liu, T. (2023). Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications. arXiv. https://doi.org/10.48550/arxiv.2311.05876

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.