CoChat

Towards Explainable AI in Agentic Retrieval-Augmented Generation: A Systematic Review

Open access
Habib, Afnan, Abdulmahmod, Osamah F., Raza, Mukhlis, Gu, Yeong Hyeon, Aydoğan, Murat, Al-antari, Mugahed A. · 2025 · 2025 9th International Artificial Intelligence and Data Processing Symposium (IDAP)

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

Abstract

The development of large language models (LLMs) is rapidly advancing, with the recent Retrieval Augmented Generation (RAG) systems and Agentic RAG systems, which combine external knowledge to improve factual accuracy and extend the autonomous reasoning processes, multi-step planning, and tool interaction to address complex tasks. The complex structure of retrieval, planning, and generation processes that the composition of the Agentic RAG systems poses considerable difficulty in matters of transparency, accountability, and trust to the user. The field of Explainable AI (XAI) in Agentic RAG is relatively unexplored, with different approaches and an absence of agreed-upon methods of evaluation. This survey provides a depth, systematic overview of the XAI methods suitable for the advanced Agentic RAG techniques, along with component-specific approaches for retrievers, planners/agents, and generators, as well as end-to-end pipeline-level explanations. Followed PRIMSA and PICO (Population, Intervention, Comparison, Outcome) guidelines, and for this review, collected data from different databases like IEEE Xplore (61 articles), PubMed (31 articles), and other databases (25 articles) from 2017 to 2025. The Rayyan AI tool is used in the process to remove duplicates and screen the articles. Initially reviewed articles based on title and abstract, followed by full text. To assess the quality of selected articles Meta Quality Appraisal Tool (MetaQAT) was used, and after screening, 39 records were excluded. In the full-text review of 78 articles, 34 articles were lacking AI relevance, thus excluded. Ultimately, 44 key articles were identified for their contribution to LLM, Agentic RAG, and Explainable AI. This review provides a comprehensive analysis of Explainable AI (XAI) techniques in Agentic Retrieval-Augmented Generation (Agentic RAG), guiding future research and advancing the fields of explainable AI (XAI), large language models (LLMs), and Agents.

Overview

Integrity signals

Registry

Crossref · 10.1109/idap68205.2025.11222281

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 18 references

Research works printed but not linked: 27

  1. Brown, 2020, Language models are few-shot learners, Advances in neural information processing systems, 33, 1877
  2. Lewis, 2020, Retrieval-augmented generation for knowledge-intensive nlp tasks, Advances in neural information processing systems, 33, 9459
  3. Cambria, 2024, Xai meets llms: A survey of the relation between explainable ai and large language models, arXiv preprint
  4. Park, Application of Med-PaLM 2 in the refinement of MIMIC-CXR labels
  5. Achiam, 2023, Gpt-4 technical report, arXiv preprint
  6. Touvron, 2023, Llama: Open and efficient foundation language models, arXiv preprint
  7. Ji, 2023, Vicunaner: Zero/few-shot named entity recognition using vicuna, arXiv preprint
  8. Kassem, 2024, Alpaca against vicuna: Using llms to uncover memorization of 1 lms, arXiv preprint
  9. Chowdhery, 2023, Palm: Scaling language modeling with pathways, Journal of Machine Learning Research, 24, 1
  10. Chung, 2024, Scaling instruction-finetuned language models, Journal of Machine Learning Research, 25, 1
  11. Anil, 2023, Palm 2 technical report, arXiv preprint
  12. Du, 2022, Glam: Efficient scaling of language models with mixture-of-experts
  13. Samo, 2024, Fine-tuning mistral 7b large language model for python query response and code generation: A parameter efficient approach, VAWKUM Transactions on Computer Sciences, 12, 205
  14. Thoppilan, 2022, Lamda: Language models for dialog applications, arXiv preprint
  15. Zhang, 2022, Opt: Open pre-trained transformer language models, arXiv preprint
  16. Gao, 2023, Retrieval-augmented generation for large language models: A survey, arXiv preprint, 2
  17. Huang, 2023, Open-source Language Models: Identifying Contributing Factors and Future Trends
  18. Wu, 2025, Talk to Right Specialists: Routing and Planning in Multi-agent System for Question Answering, arXiv preprint
  19. Madaan, 2023, Self-refine: Iterative refinement with selffeedback, Advances in Neural Information Processing Systems, 36, 46534
  20. Han, Retrieval-Augmented Generation with Graphs (GraphRAG), arXiv 2024. arXiv preprint arXiv
  21. Dawid, 2025, Agentic workflows for economic research: Design and implementation, arXiv preprint
  22. 10.3321/j.issn:0529-6579.2007.z1.029 · doi:10.3321/j.issn:0529-6579.2007.z1.029
  23. Yao, 2023, React: Synergizing reasoning and acting in language models, 2023
  24. Zhang, 2024, LLMExplainer: Large language model based bayesian inference for graph explanation generation, arXiv preprint
  25. Zheng, 2024, F- Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI
  26. Salih, 2025, Re-Visiting Explainable AI Evaluation Metrics to Identify The Most Informative Features, arXiv preprint
  27. Qureshi, 2025, Explainability in Action: A MetricDriven Assessment of Five XAI Methods for Healthcare Tabular Models, medRxiv, 2025.05. 20.25327976

Cited by

See all 3

Metadata

Authors
Habib, A., Abdulmahmod, O. F., Raza, M., Gu, Y. H., Aydoğan, M., & Al-antari, M. A.
Affiliations
  • Sejong University (KR)
  • Fırat University (TR)
  • College of AI Convergence, Sejong University,Department of Artificial Intelligence and Data Science,Seoul,Republic of Korea
Venue
2025 9th International Artificial Intelligence and Data Processing Symposium (IDAP)
Publisher
IEEE
Published
2025-09-06
Pages
1-8
OpenAlex
W4416183802
Cited by
3
References
16
Citations by year
2026: 3
FWCI
0.29
Type
journal article
Language
en
Open access
Locations
  • International Artificial Intelligence and Data Processing Symposium (IDAP) (publishedVersion)
  • DSpace - FIRAT (Fırat University) (submittedVersion, cc-by-nc-nd)
Funders
  • Korea government(Ministry of Science and ICT) (IITP-2025-RS-2024-00437191)
  • National Research Foundation of Korea (NRF)
  • Scientific and Technological Research Council of Turkey (TUBITAK) (123N325)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
explainable AI, agentic retrieval-augmented generation, large language models, AI agents, Retriever, generators, trust and transparency, multi-stage planning, tool interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Towards Explainable AI in Agentic Retrieval-Augmented Generation: A Systematic Review. https://app.cochat.ai/papers/10.1109/idap68205.2025.11222281
The paper
Habib, A., Abdulmahmod, O. F., Raza, M., Gu, Y. H., Aydoğan, M., & Al-antari, M. A. (2025). Towards Explainable AI in Agentic Retrieval-Augmented Generation: A Systematic Review. 2025 9th International Artificial Intelligence and Data Processing Symposium (IDAP). https://doi.org/10.1109/idap68205.2025.11222281

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.