Reliable Graph-RAG for Codebases: AST-Derived Graphs vs LLM-Extracted Knowledge Graphs
Open accessAbstract
Retrieval-Augmented Generation for software engineering often relies on vector similarity search, which captures topical similarity but can fail on multi-hop architectural reasoning such as controller to service to repository chains, interface-driven wiring, and inheritance. This paper benchmarks three retrieval pipelines on Java codebases (Shopizer, with additional runs on ThingsBoard and OpenMRS Core): (A) vector-only No-Graph RAG, (B) an LLM-generated knowledge graph RAG (LLM-KB), and (C) a deterministic AST-derived knowledge graph RAG (DKB) built with Tree-sitter and bidirectional traversal. Using 15 architecture and code-tracing queries per repository, we measure indexing time, query latency, corpus coverage, cost, and answer correctness. DKB builds its graph in seconds, while LLM-KB requires much longer graph generation. LLM-KB also shows indexing incompleteness: on Shopizer, 377 files are skipped or missed, reducing embedded chunk coverage and graph size compared to DKB. End-to-end cost is modest for DKB relative to the vector-only baseline but much higher for LLM-KB, especially as repository scale increases. Query latency is similar for No-Graph and DKB, while LLM-KB is slower and more variable. On the Shopizer question suite, DKB achieves the highest correctness, LLM-KB is close behind, and the vector-only baseline performs worst on upstream architectural queries and has the highest hallucination risk. Overall, deterministic AST-derived graphs provide more reliable coverage and multi-hop grounding than LLM-extracted graphs at substantially lower indexing cost.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
External stores can be updated immediately without retraining and can maintain explicit provenance and versioning.
“External stores, by contrast, can be updated immediately without retraining and can maintain explicit provenance and versioning [Oelen et al., 2025, Chinthareddy, 2026].”
From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Chinthareddy, 2026
The Introduction (§1) describes retrieving codebase context, and §12.2 records pinned repository commits for experimental reproducibility. Neither passage establishes that external stores can be updated immediately without retraining or maintain explicit provenance and versioning. Searches for “retraining,” “provenance,” and graph updates found no passage addressing those capabilities. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 24
- A. Vaswani et al., "Attention Is All You Need," in Proc. NeurIPS, 10.48550/arXiv.1706.03762, 2017. · DOI · Paper page
- P. Lewis et al., "Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks," arXiv preprint arXiv:2005.11401, 2020. · arXiv
- V. Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering," in Proc. EMNLP ,10.18653/v1/2020.emnlp- main.550, 2020. · DOI · Paper page
- G. Izacard et al., "Few-shot Learning with Retrieval Augmented Lan- guage Models," arXiv preprint arXiv:2208.03299, 2022. · arXiv
- Y. Gao et al., "Retrieval-Augmented Generation for Large Language Models: A Survey," arXiv preprint arXiv:2312.10997, 2023. · arXiv
- D. Edge et al., "From Local to Global: A Graph RAG Approach to Query-Focused Summarization," arXiv preprint arXiv:2404.16130, 2024. · arXiv
- S. Knollmeyer, O. Caymazer, and D. Grossmann, "Document GraphRAG: Knowledge Graph Enhanced Retrieval Augmented Gener- ation for Document Question Answering Within the Manufacturing Do- main," Electronics,10.3390/electronics14112102 vol. 14, no. 11, p. 2102, 2025. · DOI
- Z. Guo et al., "LightRAG: Simple and Fast Retrieval-Augmented Gen- eration," arXiv preprint arXiv:2410.05779, 2024. · arXiv
- Yilin Zhang and Xinran Zhao and Zora Zhiruo Wang and Chenyang Yang and Jiayi Wei and Tongshuang Wu, "cAST: Enhancing Code Retrieval-Augmented Generation with Structural Chunking via Ab- stract Syntax Tree," arXiv preprint arXiv:2506.15655, 2025. · arXiv
- Y. Tao et al., "Retrieval-Augmented Code Generation: A Sur- vey with Focus on Repository-Level Approaches," arXiv preprint arXiv:2510.04905, 2025. · arXiv
- Zezhou Yang and Sirong Chen and Cuiyun Gao and Zhenhao Li and Xing Hu and Kui Liu and Xin Xia, "An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportuni- ties," arXiv:2501.13742, 2025. · arXiv
- M. Chen et al., "Evaluating Large Language Models Trained on Code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
- J. Austin et al., "Program Synthesis with Large Language Models," arXiv preprint arXiv:2108.07732, 2021. · arXiv
- C. E. Jimenez et al., "SWE-bench: Can Language Models Resolve Real- World GitHub Issues?" in Proc. ICLR arxiv preprint arXiv:2310.06770, 2024. · arXiv
- T. Liu, C. Xu, and J. McAuley, "RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems," in Proc. ICLR, arxiv preprint arXiv:2306.03091, 2024. · arXiv
- F. Zhang et al., "RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation (introducing RepoEval)," arXiv preprint arXiv:2303.12570, 2023. · arXiv
- S. Ouyang et al., "RepoGraph: Enhancing Repository-Level Code Generation with Graph-Based Retrieval," in Proc. ICLR, arXiv:2410.14684, 2025. · arXiv
- Hongyuan Tao and Ying Zhang and Zhenhao Tang and Hongen Peng and Xukun Zhu and Bingchang Liu and Yingguang Yang and Ziyin Zhang and Zhaogui Xu and Haipeng Zhang and Linchao Zhu and Rui Wang and Hang Yu and Jianguo Li and Peng Di, "Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Tasks," OpenReview preprint, 2025 arXiv:2505.16901. · arXiv
- Qinggang Zhang and Shengyuan Chen and Yuanchen Bei and Zheng Yuan and Huachi Zhou and Zijin Hong and Hao Chen and Yilin Xiao and Chuang Zhou and Junnan Dong and Yi Chang and Xiao Huang, "A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models," OpenReview/arXiv preprint arXiv:2501.13958, 2025. · arXiv
- Aditi Singh and Abul Ehtesham and Saket Kumar and Tala Talaei Khoei, "Agentic Retrieval-Augmented Generation: A Survey on Agen- tic RAG," arXiv preprint arXiv:2501.09136, 2025. · arXiv
- Damian Gnieciak and Tomasz Szandala,"Large Language Models Ver- sus Static Code Analysis Tools: A Systematic Benchmark for Vulnera- bility Detection," arXiv preprint arXiv:2508.04448, 2025. · arXiv
- S. Geng et al., "Generating Structured Outputs from Language Models: Benchmark and Studies," arXiv preprint arXiv:2501.10868, 2025. DOI: 10.48550/arXiv.2501.10868. · DOI · Paper page
- B. Agarwal, I. Joshi, and V. Rojkova, "Think Inside the JSON: Rein- forcement Strategy for Strict LLM Schema Adherence," arXiv preprint arXiv:2502.14905, 2025. DOI: 10.48550/arXiv.2502.14905. · DOI
- J. Hayes et al., "Measuring memorization in language models via prob- abilistic extraction," arXiv preprint arXiv:2410.19482, 2024. · arXiv
Cited by
Sorted by citations
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1 · see citing works
- Authors
- Chinthareddy, M. R.
- Corresponding author
- Chinthareddy, Manideep Reddy
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2601.08773
- OpenAlex
- W7124132963
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- retrieval-augmented generation, multi-hop reasoning, code tracing, vector similarity search, query response time, hallucination risk, ThingsBoard
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
