The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
KnowledgeNavigator: leveraging large language models for enhanced reasoning over knowledge graph
Abstract
Large language models have achieved outstanding performance on various downstream tasks with their advanced understanding of natural language and zero-shot capability. However, they struggle with knowledge constraints, particularly in tasks requiring complex reasoning or extended logical sequences. These limitations can affect their performance in question answering by leading to inaccuracies and hallucinations. This paper proposes a novel framework called KnowledgeNavigator that leverages large language models on knowledge graphs to achieve accurate and interpretable multi-hop reasoning. Especially with an analysis-retrieval-reasoning process, KnowledgeNavigator searches the optimal path iteratively to retrieve external knowledge and guide the reasoning to reliable answers. KnowledgeNavigator treats knowledge graphs and large language models as flexible components that can be switched between different tasks without additional costs. Experiments on three benchmarks demonstrate that KnowledgeNavigator significantly improves the performance of large language models in question answering and outperforms all large language models-based baselines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“KnowledgeNavigator [282] performs retrieval on KG through iterative filtering of relations with respect to core entities, so as to obtain relevant triplets.”
From Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024) · cites this paper as [282]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications
Ji, S., Pan, S., Cambria, E., Marttinen, P., & Yu, P. S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3070843 · Cited by 2,947
Embedding Entities and Relations for Learning and Inference in Knowledge Bases
Yang, B., Yih, W.-T., He, X., Gao, J., & Deng, L. (2014) · arXiv · doi:10.48550/arxiv.1412.6575 · Cited by 2,252
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.10403 · Cited by 834
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534 · cited 2×
“…have shown that LLMs suffer from hallucination and knowledge limitation, including outdated or incorrect facts, and lack of specialized knowledge [Zhang et al. , 2023; Martino et al. , 2023; Figure 1: LLM suffers from hallucination and knowledge limitation, which can be solved with external…”
Temporal Knowledge Graph Completion: A Survey
Cai, B., Xiang, Y., Gao, L., Zhang, H., Li, Y., & Li, J. (2023) · Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/734 · Cited by 133 · cited 1×
“For these tasks, the process involves retrieving correct and real-time knowledge relevant to the question, constructing prompts, and then feeding these prompts to the LLM for analysis or summary, as demonstrated in Figure 1.”
Jiang, J., Zhou, K., Zhao, W. X., & Wen, J.-R. (2022) · arXiv · doi:10.48550/arxiv.2212.00959 · Cited by 105
Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
Baek, J., Aji, A., & Saffari, A. (2023) · Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023) · doi:10.18653/v1/2023.matching-1.7 · Cited by 51 · cited 1×
“Knowled-geNavigator surpasses other research using the same back-bone LLM for knowledge retrieval and reasoning, such as TOG and StructGPT, on MetaQA.”
RHO: Reducing Hallucination in Open-domain Dialogues with Knowledge Grounding
Ji, Z., Liu, Z., Lee, N., Yu, T., Wilie, B., Zeng, M., & Fung, P. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.275 · Cited by 37
Research works printed but not linked: 35
- Bai J, Bai S, Chu Y, Cui Z, Dang K, Deng X, Fan Y, Ge W, Han Y, Huang F et al (2023) Qwen technical report. arXiv preprint arXiv:2309.16609. Accessed 07 Dec 2023 · arXiv
- Martino A, Iannelli M, Truong C (2023) Knowledge injection to counter large language model (llm) hallucination. European Semantic Web Conference. Springer, New York, pp 182–185
- Chen W, Yan-yi L, Tie-zheng G, Da-peng L, Tao H, Zhi L, Qing-wen Y, Hui-han W, Ying-you W (2024) Systems engineering issues for industry applications of large language model. Appl Soft Comput 151:111165 · DOI
- Creswell A, Shanahan M, Higgins I (2022) Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712. Accessed 10 June 2023 · arXiv
- Zhu X, Li Z, Wang X, Jiang X, Sun P, Wang X, Xiao Y, Yuan NJ (2024) Multi-modal knowledge graph construction and application: a survey. IEEE Trans Knowl Data Eng 36(02):715–735 · DOI · Paper page
- Dong G, Zhao J, Hui T, Guo D, Wang W, Feng B, Qiu Y, Gongque Z, He K, Wang Z et al (2023) Revisit input perturbation problems for llms: a unified robustness evaluation framework for noisy slot filling task. CCF International Conference on Natural Language Processing and Chinese Computing. Springer, New York, pp 682–694
- Moiseev F, Dong Z, Alfonseca E, Jaggi M (2022) Skill: structured knowledge infusion for large language models. arXiv preprint arXiv:2205.08184. Accessed 10 Oct 2023 · DOI · Paper page
- Tian L, Zhou X, Wu Y-P, Zhou W-T, Zhang J-H, Zhang T-S (2022) Knowledge graph and knowledge reasoning: A systematic review. Journal of Electronic Science and Technology 20(2):100159 · DOI
- Chakraborty N, Lukovnikov D, Maheshwari G, Trivedi P, Lehmann J, Fischer A (2021) Introduction to neural network-based question answering over knowledge graphs. Wiley Interdiscip Rev Data Min Knowl Discov 11(3):1389 · DOI
- Berant J, Chou A, Frostig R, Liang P (2013) Semantic parsing on freebase from question-answer pairs. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. pp. 1533–1544
- Yih W-t, He X, Meek C (2014) Semantic parsing for single-relation question answering. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 643–648 · DOI · Paper page
- Krötzsch M, Marx M, Ozaki A, Thost V (2018) Attributed description logics: Reasoning on knowledge graphs. In: IJCAI, pp. 5309–5313 · DOI
- Xiao Z, Xing H, Qu R, Feng L, Luo S, Dai P, Zhao B, Dai Y (2024) Densely knowledge-aware network for multivariate time series classification. IEEE Trans Syst, Man, Cybern Syst 54(4):2192–2204 · DOI · Paper page
- Xiao Z, Xing H, Zhao B, Qu R, Luo S, Dai P, Li K, Zhu Z (2023) Deep contrastive representation learning with self-distillation. IEEE Trans Emerg Topics Comput Intell 8(1):3–15 · DOI
- Huang X, Zhang J, Li D, Li P (2019) Knowledge graph embedding based question answering. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. pp. 105–113 · DOI · Paper page
- Liu W, Zhao Q, Piao S, Wang C, Kong Q, An T (2017) Endo-sirna deficiency results in oocyte maturation failure and apoptosis in porcine oocytes. Reprod Fertil Dev 29(11):2168–2174 · doi:10.1071/rd16498
- Xiong W, Yu M, Chang S, Guo X, Wang WY (2019) Improving question answering over incomplete kbs with knowledge-aware reader. arXiv preprint arXiv:1905.07098. Accessed 07 Jan 2023 · DOI
- Das R, Dhuliawala S, Zaheer M, Vilnis L, Durugkar I, Krishnamurthy A, Smola A, McCallum A (2017) Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. arXiv preprint arXiv:1711.05851. Accessed 09 June 2023 · arXiv
- Cincovic J, Jovanovic L, Nikolic B, Bacanin N (2024) Neurodegenerative condition detection using modified metaheuristic for attention based recurrent neural networks and extreme gradient boosting tuning. IEEE Access 12:26719–26734 · DOI
- Pavlov-Kagadejev M, Jovanovic L, Bacanin N, Deveci M, Zivkovic M, Tuba M, Strumberger I, Pedrycz W (2024) Optimizing long-short-term memory models via metaheuristics for decomposition aided wind energy generation forecasting. Artif Intell Rev 57(3):45 · DOI
- Pan S, Luo L, Wang Y, Chen C, Wang J, Wu X (2023) Unifying large language models and knowledge graphs: a roadmap. arXiv preprint arXiv:2306.08302. Accessed 14 July 2023 · arXiv
- Feng S, Balachandran V, Bai Y, Tsvetkov Y (2023) Factkb: Generalizable factuality evaluation using language models enhanced with factual knowledge. arXiv preprint arXiv:2305.08281 · DOI
- Yu D, Zhu C, Yang Y, Zeng M (2022) Jaket: joint pre-training of knowledge graph and language understanding. Proc AAAI Conf Artif Intell. 36:11630–11638
- Zhang Y, Dai H, Kozareva Z, Smola A, Song L (2018) Variational reasoning for question answering with knowledge graph. Proc AAAI Conf Artif Intell. 32: 6069–6076 · DOI · Paper page
- Yih W-t, Richardson M, Meek C, Chang M-W, Suh J (2016) The value of semantic parse labeling for knowledge base question answering. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 201–206 · DOI · Paper page
- Xu K, Lai Y, Feng Y, Wang Z (2019) Enhancing key-value memory neural networks for knowledge based question answering. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp 2937–2947 · DOI · Paper page
- Sun H, Dhingra B, Zaheer M, Mazaitis K, Salakhutdinov R, Cohen WW (2018) Open domain question answering using early fusion of knowledge bases and text. arXiv preprint arXiv:1809.00782. Accessed 09 Sept 2023 · DOI · Paper page
- Baek J, Aji AF, Saffari A (2023) Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136. Accessed 10 Aug 2023 · DOI · Paper page
- Sun J, Xu C, Tang L, Wang S, Lin C, Gong Y, Shum H-Y, Guo J (2023) Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.07697. Accessed 11 Oct 2023 · arXiv
- Saxena A, Tripathi A, Talukdar P (2020) Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4498–4507 · DOI
- He G, Lan Y, Jiang J, Zhao WX, Wen J-R (2021) Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In: Proceedings of the 14th ACM International Conference on Web Search and Data Mining. pp. 553–561 · DOI · Paper page
- Shi J, Cao S, Hou L, Li J, Zhang H (2021) Transfernet: An effective and transparent framework for multi-hop question answering over relation graph. arXiv preprint arXiv:2104.07302 · DOI
- Das R, Godbole A, Naik A, Tower E, Zaheer M, Hajishirzi H, Jia R, McCallum A (2022) Knowledge base question answering by case-based reasoning over subgraphs. In: International Conference on Machine Learning, pp. 4777–4793. PMLR. Accessed 21 Oct 2023
- Jiang J, Zhou K, Dong Z, Ye K, Zhao WX, Wen J-R (2023) Structgpt: A general framework for large language model to reason over structured data. arXiv preprint arXiv:2305.09645 · DOI
- Touvron H, Martin L, Stone K, Albert P, Almahairi A, Babaei Y, Bashlykov N, Batra S, Bhargava P, Bhosale S, et al (2023) Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 · arXiv
Cited by
Zhao, S., Yang, Y., Wang, Z., He, Z., Qiu, L. K., & Qiu, L. (2024) · arXiv · doi:10.48550/arxiv.2409.14924 · Cited by 180
Retrieval-Augmented Generation with Graphs (GraphRAG)
Han, H., Wang, Y., Shomer, H., Guo, K., Ding, J., Lei, Y., Halappanavar, M., Rossi, R. A., Mukherjee, S., Tang, X., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.00309 · Cited by 34
Self-citation
Metadata
- Authors
- Guo, T., Yang, Q., Wang, C., Liu, Y., Li, P., Tang, J., Li, D., & Wen, Y.
- ORCID
- Wen, Yingyou
- Venue
- Complex & Intelligent Systems
- Published
- 2024-07-02
- Volume
- 10
- Issue
- 5
- Pages
- 7063-7076
- ISSN
- 2199-4536
- 2198-6053
- Cited by
- 54
- References
- 10
- Language
- en
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Funders
- Shenyang Science and Technology Plan Project (RC210469)
- Liaoning Provincial Science and Technology Innovation Project in the Field of Artificial Intelligence (2023JH26/10100005)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: KnowledgeNavigator: leveraging large language models for enhanced reasoning over knowledge graph. https://app.cochat.ai/papers/10.1007/s40747-024-01527-8
- The paper
- Guo, T., Yang, Q., Wang, C., Liu, Y., Li, P., Tang, J., Li, D., & Wen, Y. (2024). KnowledgeNavigator: leveraging large language models for enhanced reasoning over knowledge graph. Complex & Intelligent Systems. https://doi.org/10.1007/s40747-024-01527-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.