CoChat
Sign Up
PreprintCited by 21Topic ModelingInformation Retrieval and Search BehaviorAdvanced Graph Neural Networks

Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning

Open access
Markovic, Vasilije, Obradovic, Lazar, Hajdu, Laszlo, Pavlovic, Jovan·2025·arXiv
DOI 10.48550/arxiv.2505.24478arXiv 2505.24478
Open this paper in CoChat — free accountOpen PDF

Abstract

Integrating Large Language Models (LLMs) with Knowledge Graphs (KGs) results in complex systems with numerous hyperparameters that directly affect performance. While such systems are increasingly common in retrieval-augmented generation, the role of systematic hyperparameter optimization remains underexplored. In this paper, we study this problem in the context of Cognee, a modular framework for end-to-end KG construction and retrieval. Using three multi-hop QA benchmarks (HotPotQA, TwoWikiMultiHop, and MuSiQue) we optimize parameters related to chunking, graph construction, retrieval, and prompting. Each configuration is scored using established metrics (exact match, F1, and DeepEval's LLM-based correctness metric). Our results demonstrate that meaningful gains can be achieved through targeted tuning. While the gains are consistent, they are not uniform, with performance varying across datasets and metrics. This variability highlights both the value of tuning and the limitations of standard evaluation measures. While demonstrating the immediate potential of hyperparameter tuning, we argue that future progress will depend not only on architectural advances but also on clearer frameworks for optimization and evaluation in complex, modular systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.24478

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Knowledge graph memory systems are well-suited for representing structured relationships between entities but struggle to model sequential events, emotional states, full-length documents, or multi-modal inputs such as images.

    “These frameworks are well-suited for representing structured relationships between entities but struggle to model sequential events, emotional states, full-length documents, or multi-modal inputs such as images.”

    From MIRIX: Multi-Agent Memory System for LLM-Based Agents (2025) · cites this paper as [20]

    The Introduction (p. 2) supports the claim that graph-based systems represent structured relationships. The paper does not say they struggle with sequential events or emotional states; its description of Cognee instead says the pipeline processes PDFs and images (Appendix A.1, p. 12), contrary to the claimed difficulty with documents and multimodal inputs. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 41

  1. [1][1] Microsoft Research AI4Science and Microsoft Azure Quantum. 'The impact of large language models on scientific discovery: a preliminary study using gpt-4'. In: arXiv preprint arXiv:2311.07361 (2023). - · arXiv
  2. [2][2] Akari Asai et al. 'Self-rag: Learning to retrieve, generate, and critique through self-reflection'. In: The Twelfth International Conference on Learning Representations . 2023. -
  3. [3][3] Matthew Barker et al. 'Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems'. In: arXiv preprint arXiv:2502.18635 (2025). - · arXiv
  4. [4][4] Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 'Neural legal judgment prediction in English'. In: arXiv preprint arXiv:1906.02059 (2019). - · arXiv
  5. [5][5] Yiqun Chen et al. 'Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning'. In: arXiv preprint arXiv:2501.15228 (2025). - · arXiv
  6. [6][6] Nurendra Choudhary and Chandan K Reddy. 'Complex logical reasoning over knowledge graphs using large language models'. In: arXiv preprint arXiv:2305.01157 (2023). - · arXiv
  7. [7][7] Darren Edge et al. 'From local to global: A graph rag approach to queryfocused summarization'. In: arXiv preprint arXiv:2404.16130 (2024). - · arXiv
  8. [8][8] Wenqi Fan et al. 'A survey on rag meeting llms: Towards retrieval-augmented large language models'. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining . 2024, pp. 6491-6501. -
  9. [9][9] Jia Fu et al. 'AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation'. In: arXiv preprint arXiv:2406.19251 (2024). - · arXiv
  10. [10][10] Yunfan Gao et al. 'Retrieval-augmented generation for large language models: A survey'. In: arXiv preprint arXiv:2312.10997 2 (2023). - · arXiv
  11. [11][11] Shailja Gupta, Rajesh Ranjan, and Surya Narayan Singh. 'A comprehensive survey of retrieval-augmented generation (rag): Evolution, current landscape and future directions'. In: arXiv preprint arXiv:2410.12837 (2024). - · arXiv
  12. [12][12] Bernal Jim´ enez Guti´ errez et al. 'Hipporag: Neurobiologically inspired long-term memory for large language models'. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems . 2024. -
  13. [13][13] Kelvin Guu et al. 'Retrieval augmented language model pre-training'. In: International conference on machine learning . PMLR. 2020, pp. 39293938. -
  14. [14][14] Haoyu Han et al. 'Retrieval-augmented generation with graphs (graphrag)'. In: arXiv preprint arXiv:2501.00309 (2024). - · arXiv
  15. [15][15] Xiaoxin He et al. 'G-retriever: Retrieval-augmented generation for textual graph understanding and question answering'. In: Advances in Neural Information Processing Systems 37 (2024), pp. 132876-132907. -
  16. [16][16] Xanh Ho et al. 'Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps'. In: arXiv preprint arXiv:2011.01060 (2020). - · arXiv
  17. [17][17] Lei Huang et al. 'A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions'. In: ACM Transactions on Information Systems 43.2 (2025), pp. 1-55. -
  18. [18][18] Kush Juvekar and Anupam Purwar. 'Introducing a new hyper-parameter for RAG: Context Window Utilization'. In: arXiv preprint arXiv:2407.19794 (2024). - · arXiv
  19. [19][19] Ernests Lavrinovics et al. 'Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective'. In: Journal of Web Semantics 85 (2025), p. 100844. -
  20. [20][20] Patrick Lewis et al. 'Retrieval-augmented generation for knowledge-intensive nlp tasks'. In: Advances in neural information processing systems 33 (2020), pp. 9459-9474. -
  21. [21][21] Junyi Li et al. 'The dawn after the dark: An empirical study on factuality hallucination in large language models'. In: arXiv preprint arXiv:2401.03205 (2024). - · arXiv
  22. [22][22] Linhao Luo et al. 'Reasoning on graphs: Faithful and interpretable large language model reasoning'. In: arXiv preprint arXiv:2310.01061 (2023). - · arXiv
  23. [23][23] Alex Mallen et al. 'When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories'. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Ed. by Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki. Toronto, Canada: Association for Computational Linguistics, July 2023, pp. 9802-9822. doi : 10.18653/v1/2023.acllong.546 . url : https://aclanthology.org/2023.acl-long.546/ . - · DOI
  24. [24][24] Shirui Pan et al. 'Unifying large language models and knowledge graphs: A roadmap'. In: IEEE Transactions on Knowledge and Data Engineering 36.7 (2024), pp. 3580-3599. -
  25. [25][25] Boci Peng et al. 'Graph retrieval-augmented generation: A survey'. In: arXiv preprint arXiv:2408.08921 (2024). - · arXiv
  26. [26][26] Fabio Petroni et al. 'Language models as knowledge bases?' In: arXiv preprint arXiv:1909.01066 (2019). - · arXiv
  27. [27][27] Adam Roberts, Colin Raffel, and Noam Shazeer. 'How much knowledge can you pack into the parameters of a language model?' In: arXiv preprint arXiv:2002.08910 (2020). - · arXiv
  28. [28][28] Karan Singhal et al. 'Toward expert-level medical question answering with large language models'. In: Nature Medicine (2025), pp. 1-8. -
  29. [29][29] Alon Talmor and Jonathan Berant. 'The web as a knowledge-base for answering complex questions'. In: arXiv preprint arXiv:1803.06643 (2018). - · arXiv
  30. [30][30] Vincent A Traag, Ludo Waltman, and Nees Jan Van Eck. 'From Louvain to Leiden: guaranteeing well-connected communities'. In: Scientific reports 9.1 (2019), pp. 1-12. -
  31. [31][31] Harsh Trivedi et al. 'MuSiQue: Multihop Questions via Single-hop Question Composition'. In: Transactions of the Association for Computational Linguistics 10 (2022), pp. 539-554. -
  32. [32][32] Ashish Vaswani et al. 'Attention is all you need'. In: Advances in neural information processing systems 30 (2017). -
  33. [33][33] Chi Wang, Xueqing Liu, and Ahmed Hassan Awadallah. 'Cost-effective hyperparameter optimization for large language model generation inference'. In: International Conference on Automated Machine Learning . PMLR. 2023, pp. 21-1. -
  34. [34][34] Yu Wang et al. 'Knowledge graph prompting for multi-document question answering'. In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 38. 17. 2024, pp. 19206-19214. -
  35. [35][35] Shijie Wu et al. 'Bloomberggpt: A large language model for finance'. In: arXiv preprint arXiv:2303.17564 (2023). - · arXiv
  36. [37][37] Zhilin Yang et al. 'HotpotQA: A dataset for diverse, explainable multihop question answering'. In: arXiv preprint arXiv:1809.09600 (2018). - · arXiv
  37. [38][38] Wen-tau Yih et al. 'The value of semantic parse labeling for knowledge base question answering'. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) . 2016, pp. 201-206. -
  38. [39][39] Jing Zhang et al. 'Neural, symbolic and neural-symbolic reasoning on knowledge graphs'. In: AI Open 2 (2021), pp. 14-35. -
  39. [40][40] Jing Zhang et al. 'Subgraph retrieval enhanced model for multi-hop knowledge base question answering'. In: arXiv preprint arXiv:2202.13296 (2022). - · arXiv
  40. [41][41] Yuyu Zhang et al. 'Variational reasoning for question answering with knowledge graph'. In: Proceedings of the AAAI conference on artificial intelligence . Vol. 32. 1. 2018. -
  41. [42][42] Penghao Zhao et al. 'Retrieval-augmented generation for ai-generated content: A survey'. In: arXiv preprint arXiv:2402.19473 (2024). · arXiv

Cited by

Sorted by citations

Related

Authors
Markovic, V., Obradovic, L., Hajdu, L., & Pavlovic, J.
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.24478
arXiv
2505.24478
OpenAlex
W4414857681
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
Keywords
knowledge graphs, large language models, hyperparameter optimization, retrieval-augmented generation, multi-hop question answering, chunking, graph construction, retrieval, prompting, evaluation metrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review