Towards Explainable AI in Agentic Retrieval-Augmented Generation: A Systematic Review
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
The development of large language models (LLMs) is rapidly advancing, with the recent Retrieval Augmented Generation (RAG) systems and Agentic RAG systems, which combine external knowledge to improve factual accuracy and extend the autonomous reasoning processes, multi-step planning, and tool interaction to address complex tasks. The complex structure of retrieval, planning, and generation processes that the composition of the Agentic RAG systems poses considerable difficulty in matters of transparency, accountability, and trust to the user. The field of Explainable AI (XAI) in Agentic RAG is relatively unexplored, with different approaches and an absence of agreed-upon methods of evaluation. This survey provides a depth, systematic overview of the XAI methods suitable for the advanced Agentic RAG techniques, along with component-specific approaches for retrievers, planners/agents, and generators, as well as end-to-end pipeline-level explanations. Followed PRIMSA and PICO (Population, Intervention, Comparison, Outcome) guidelines, and for this review, collected data from different databases like IEEE Xplore (61 articles), PubMed (31 articles), and other databases (25 articles) from 2017 to 2025. The Rayyan AI tool is used in the process to remove duplicates and screen the articles. Initially reviewed articles based on title and abstract, followed by full text. To assess the quality of selected articles Meta Quality Appraisal Tool (MetaQAT) was used, and after screening, 39 records were excluded. In the full-text review of 78 articles, 34 articles were lacking AI relevance, thus excluded. Ultimately, 44 key articles were identified for their contribution to LLM, Agentic RAG, and Explainable AI. This review provides a comprehensive analysis of Explainable AI (XAI) techniques in Agentic Retrieval-Augmented Generation (Agentic RAG), guiding future research and advancing the fields of explainable AI (XAI), large language models (LLMs), and Agents.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Although recent reviews have examined the capabilities, architectures, and clinical applications of agentic AI in healthcare, explainability has been largely treated as a conceptual requirement rather than an operational property of these systems [1,4,12].”
From Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review (2026) · cites this paper as [12]
In the citing paper
“These dimensions were categorized according to established and previously validated XAI taxonomies [12,17,18], which were adapted to the context of agentic AI systems as a provisional analytic schema to enable consistent classification across heterogeneous implementations.”
From Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review (2026) · cites this paper as [12]
Citations
Cites
Transformer Interpretability Beyond Attention Visualization
Chefer, H., Gur, S., & Wolf, L. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00084 · Cited by 903
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534
Jeong, S., Baek, J., Cho, S., Hwang, S. J., & Park, J. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.389 · Cited by 299
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278 · cited 2×
“DoReMi Framework, Rong et al. (2023) [ 37 ] Human-centered Aligns to user preferences.”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.523 · Cited by 161
MAIN-RAG: Multi-Agent Filtering Retrieval-Augmented Generation
Chang, C.-Y., Jiang, Z., Rakesh, V., Pan, M., Yeh, C. M., Wang, G., Hu, M., Xu, Z., Zheng, Y. L., Das, M., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.131 · Cited by 157
Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters
Wang, B., Min, S., Deng, X., Shen, J., Wu, Y., Zettlemoyer, L., & Sun, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.153 · Cited by 120
HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generation
Liu, P., Liu, X., Yao, R., Liu, J., Meng, S., Wang, D., & Ma, J. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3754761 · Cited by 111
A Robust Governance for the AI Act: AI Office, AI Board, Scientific Panel, and National Authorities
Novelli, C., Hacker, P., Morley, J., Trondal, J., & Floridi, L. (2024) · European Journal of Risk Regulation · doi:10.1017/err.2024.57 · Cited by 100
Naveed, S., Stevens, G., & Robin-Kern, D. (2024) · Applied Sciences · doi:10.3390/app142311288 · Cited by 89
Research works printed but not linked: 27
- Brown, 2020, Language models are few-shot learners, Advances in neural information processing systems, 33, 1877
- Lewis, 2020, Retrieval-augmented generation for knowledge-intensive nlp tasks, Advances in neural information processing systems, 33, 9459
- Cambria, 2024, Xai meets llms: A survey of the relation between explainable ai and large language models, arXiv preprint
- Park, Application of Med-PaLM 2 in the refinement of MIMIC-CXR labels
- Achiam, 2023, Gpt-4 technical report, arXiv preprint
- Touvron, 2023, Llama: Open and efficient foundation language models, arXiv preprint
- Ji, 2023, Vicunaner: Zero/few-shot named entity recognition using vicuna, arXiv preprint
- Kassem, 2024, Alpaca against vicuna: Using llms to uncover memorization of 1 lms, arXiv preprint
- Chowdhery, 2023, Palm: Scaling language modeling with pathways, Journal of Machine Learning Research, 24, 1
- Chung, 2024, Scaling instruction-finetuned language models, Journal of Machine Learning Research, 25, 1
- Anil, 2023, Palm 2 technical report, arXiv preprint
- Du, 2022, Glam: Efficient scaling of language models with mixture-of-experts
- Samo, 2024, Fine-tuning mistral 7b large language model for python query response and code generation: A parameter efficient approach, VAWKUM Transactions on Computer Sciences, 12, 205
- Thoppilan, 2022, Lamda: Language models for dialog applications, arXiv preprint
- Zhang, 2022, Opt: Open pre-trained transformer language models, arXiv preprint
- Gao, 2023, Retrieval-augmented generation for large language models: A survey, arXiv preprint, 2
- Huang, 2023, Open-source Language Models: Identifying Contributing Factors and Future Trends
- Wu, 2025, Talk to Right Specialists: Routing and Planning in Multi-agent System for Question Answering, arXiv preprint
- Madaan, 2023, Self-refine: Iterative refinement with selffeedback, Advances in Neural Information Processing Systems, 36, 46534
- Han, Retrieval-Augmented Generation with Graphs (GraphRAG), arXiv 2024. arXiv preprint arXiv
- Dawid, 2025, Agentic workflows for economic research: Design and implementation, arXiv preprint
- 10.3321/j.issn:0529-6579.2007.z1.029 · doi:10.3321/j.issn:0529-6579.2007.z1.029
- Yao, 2023, React: Synergizing reasoning and acting in language models, 2023
- Zhang, 2024, LLMExplainer: Large language model based bayesian inference for graph explanation generation, arXiv preprint
- Zheng, 2024, F- Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI
- Salih, 2025, Re-Visiting Explainable AI Evaluation Metrics to Identify The Most Informative Features, arXiv preprint
- Qureshi, 2025, Explainability in Action: A MetricDriven Assessment of Five XAI Methods for Healthcare Tabular Models, medRxiv, 2025.05. 20.25327976
Cited by
Pérez, J., González, I., Imam, N., & Carvajal, J. C. (2026) · Mathematics · doi:10.3390/math14081263 · Cited by 1
“Work focused on explainable agentic RAG further highlights that interpretability and accountability have become first-order requirements as these systems move toward higher impact decisions [6].”
Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review
Collaco, B. G., Prabha, S., Gomez-Cabello, C. A., Haider, S. A., Genovese, A., Wood, N. G., Gopala, N., Raman, R., Hester, E. O., & Forte, A. J. (2026) · Bioengineering · doi:10.3390/bioengineering13050513 · Cited by 3
“Although recent reviews have examined the capabilities, architectures, and clinical applications of agentic AI in healthcare, explainability has been largely treated as a conceptual requirement rather than an operational property of these systems [1,4,12].”
InfluentialWulf, J., Meierhofer, J., & Dömer, M. (2026) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds70563.2026.00013
“Explainability methods, such as component-specific explanations that trace LLM answers back to external sources [20], [21], are essential for provenance-aware oversight.”
Metadata
- Authors
- Habib, A., Abdulmahmod, O. F., Raza, M., Gu, Y. H., Aydoğan, M., & Al-antari, M. A.
- Affiliations
- Sejong University (KR)
- Fırat University (TR)
- College of AI Convergence, Sejong University,Department of Artificial Intelligence and Data Science,Seoul,Republic of Korea
- ORCID
- Venue
- 2025 9th International Artificial Intelligence and Data Processing Symposium (IDAP)
- Publisher
- IEEE
- Published
- 2025-09-06
- Pages
- 1-8
- OpenAlex
- W4416183802
- Cited by
- 3
- References
- 16
- Citations by year
- 2026: 3
- FWCI
- 0.29
- Type
- journal article
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- International Artificial Intelligence and Data Processing Symposium (IDAP) (publishedVersion)
- DSpace - FIRAT (Fırat University) (submittedVersion, cc-by-nc-nd)
- Funders
- Korea government(Ministry of Science and ICT) (IITP-2025-RS-2024-00437191)
- National Research Foundation of Korea (NRF)
- Scientific and Technological Research Council of Turkey (TUBITAK) (123N325)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Keywords
- explainable AI, agentic retrieval-augmented generation, large language models, AI agents, Retriever, generators, trust and transparency, multi-stage planning, tool interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards Explainable AI in Agentic Retrieval-Augmented Generation: A Systematic Review. https://app.cochat.ai/papers/10.1109/idap68205.2025.11222281
- The paper
- Habib, A., Abdulmahmod, O. F., Raza, M., Gu, Y. H., Aydoğan, M., & Al-antari, M. A. (2025). Towards Explainable AI in Agentic Retrieval-Augmented Generation: A Systematic Review. 2025 9th International Artificial Intelligence and Data Processing Symposium (IDAP). https://doi.org/10.1109/idap68205.2025.11222281
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.