Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index; 50 of the 68 sources its claims cite were checked for support.

Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review

Open access

Abstract

BACKGROUND: Agentic artificial intelligence (AI) systems, characterized by autonomous goal-directed behavior, multi-step reasoning, task decomposition, and tool use, are increasingly proposed for healthcare applications. However, their autonomy raises concerns regarding transparency, accountability, and human oversight. While explainable AI (XAI) has been widely studied in traditional predictive models, less is known about how explainability is implemented within agentic architectures. OBJECTIVE: To map the emerging literature on explainable agentic AI (XAAI) in healthcare and characterize the types, scope, and forms of explainability used in these systems. METHODS: A scoping review was conducted following PRISMA-ScR guidelines. PubMed, Embase, IEEE Xplore, and ACM Digital Library were searched through November 2025. Eligible studies described healthcare-related agentic AI systems incorporating explicit explainability mechanisms. Data were extracted on system architecture, explainability type (intrinsic, post hoc, hybrid), explanation scope (local, global), explanation form, and reported clinical outcomes. RESULTS: Nine studies met the inclusion criteria. All systems demonstrated core agentic features, including autonomy, task decomposition, and tool integration, often within multi-agent frameworks. Explainability was predominantly intrinsic and workflow-native, typically delivered through textual reasoning traces and example-based grounding in retrieved clinical evidence. Feature-based and global explanations were comparatively rare and largely confined to hybrid architectures. Across domains including radiology, neurology, psychiatry, and biomedical research, XAAI systems were reported to improve performance and interpretability relative to baseline models in the included studies. However, these findings were derived from heterogeneous, predominantly experimental or retrospective studies, and structured human-in-the-loop oversight was infrequently described. CONCLUSIONS: Current XAAI systems appear to emphasize process transparency and evidence grounding rather than mechanistic model-level attribution. The available evidence remains limited and heterogeneous, and findings should be interpreted as early trends rather than established characteristics. Further progress will require standardized evaluation frameworks, clearer reporting of oversight mechanisms, and validation in real-world clinical settings to support safe and trustworthy integration of agentic AI into healthcare practice.

Overview

CoChat Summary

TL;DR

This PRISMA-ScR scoping review maps nine studies of explainable agentic AI (XAAI) in healthcare, finding that explainability is predominantly intrinsic, workflow-native, and delivered through textual reasoning traces plus evidence grounding. The evidence base is early, heterogeneous, and largely experimental or retrospective, with little operationalized human-in-the-loop oversight.

Key findings

  • Nine studies met inclusion criteria; all systems showed core agentic features—autonomy, task decomposition, and tool integration—often in multi-agent frameworks. Explainability was predominantly intrinsic and workflow-native, typically via textual reasoning traces and example-based grounding in retrieved clinical evidence; feature-based and global explanations were comparatively rare and largely confined to hybrid architectures.
  • Reported outcomes were generally favorable versus single-agent LLM, traditional ML, or non-reasoning baselines across radiology, neurology, psychiatry, and biomedical tasks. Specific reported numbers include DSM5AgentFlow F1 scores up to 77%, CASSIA annotation accuracy improvement of 12–41% across large-scale cell populations, and ADAM-1 higher and more stable F1 scores than traditional machine learning models.
  • No included study reported prospective clinical deployment. Structured human-in-the-loop oversight was infrequently described, and none clearly operationalized how clinicians or domain experts would supervise, validate, or intervene in autonomous agentic workflows.
  • The authors characterize current XAAI as emphasizing process transparency and evidence traceability over mechanistic model-level attribution, with findings to be interpreted as early trends rather than established characteristics.

Methods

  • Scoping review conducted following PRISMA-ScR guidelines, with protocol registered in the Open Science Framework. PubMed, Embase, IEEE Xplore, and ACM Digital Library were searched on 24 November 2025; no restriction on publication date or language.
  • Eligibility required healthcare-related agentic AI systems with autonomous, goal-directed execution, explicit multi-step reasoning or task decomposition, and explicit explainability/transparency features. Empirical research, simulations, prototypes, architecture papers, algorithmic evaluations, and qualitative studies were included; non-health, non-agentic, non-explainable, or insufficiently detailed studies were excluded.
  • Two independent reviewers extracted data on system architecture, explainability type (intrinsic, post hoc, hybrid), scope (local, global), form (feature-based, textual, example-based), specificity, and reported outcomes. Synthesis was descriptive and narrative; no formal meta-analysis was performed due to substantial heterogeneity.

Limitations

  • The paper states that the number of included studies is modest, most systems were evaluated experimentally or as proof-of-concept rather than in prospective clinical deployments, and substantial heterogeneity in tasks, architectures, datasets, and reporting limits direct comparison.
  • The paper states that reported outcomes were extracted as described by study authors and were not independently verified or standardized; inclusion of preprints and non-peer-reviewed sources may introduce variability in rigor; the adapted XAAI explainability taxonomy has not undergone formal validation; and requiring explicit explainability may have introduced selection bias toward systems already framed as explainable.
  • Evident from the text: no meta-analysis was possible, no included study reported prospective clinical deployment, and some trust/usability claims—such as DSM5AgentFlow’s transparent and auditable diagnostic assessments—were not formally evaluated in user-centered studies.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/bioengineering13050513

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Conceptual transition from traditional “black-box” AI to trustworthy AI through explainability. Created in BioRender. Collaco, B. (2026) https://BioRender.com/3eyye5r (accessed on 12 February 2026).
Figure 1 Conceptual transition from traditional “black-box” AI to trustworthy AI through explainability. Created in BioRender. Collaco, B. (2026) https://BioRender.com/3eyye5r (accessed on 12 February 2026).CC BY · PubMed Central
Figure 2 Proposed taxonomy of explainable agentic artificial intelligence (XAAI) in healthcare. Created in BioRender. Collaco, B. (2026) https://BioRender.com/3eyye5r (accessed on 12 February 2026).
Figure 2 Proposed taxonomy of explainable agentic artificial intelligence (XAAI) in healthcare. Created in BioRender. Collaco, B. (2026) https://BioRender.com/3eyye5r (accessed on 12 February 2026).CC BY · PubMed Central
Figure 3 PRISMA flow chart of included studies.
Figure 3 PRISMA flow chart of included studies.CC BY · PubMed Central
Figure 4 Core architectural components of XAAI in healthcare. Created in BioRender. Collaco, B. (2026) https://BioRender.com/3eyye5r (accessed on 12 February 2026).
Figure 4 Core architectural components of XAAI in healthcare. Created in BioRender. Collaco, B. (2026) https://BioRender.com/3eyye5r (accessed on 12 February 2026).CC BY · PubMed Central
Figure 5 Trustworthy explainable healthcare agentic AI workflow. Created in BioRender. Collaco, B. (2026) https://BioRender.com/3eyye5r (accessed on 12 February 2026).
Figure 5 Trustworthy explainable healthcare agentic AI workflow. Created in BioRender. Collaco, B. (2026) https://BioRender.com/3eyye5r (accessed on 12 February 2026).CC BY · PubMed Central

Verification

Claims the paper makes

17 Valid · 2 Not valid · 7 Not checked · 64 Own results and interpretations

48 more claims · 8 with their cited sources checked

Create a free account to see all 90 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 23
  • Probably not a substantial claim. 18

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 40 references

Research works printed but not linked: 7

  1. Blake, H. (2026, April 23). Transparency and Explainability in AI: Ensuring Trust and Understanding in Autonomous Decision-Making Systems. Available online: https://www.researchgate.net/publication/387419097_Transparency_and_Explainability_in_AI_Ensuring_Trust_and_Understanding_in_Autonomous_Decision-Making_Systems.
  2. Inoue, Y., Song, T., Wang, X., Luna, A., and Fu, T. (2025). DruGagent: Multi-Agent Large Language Model-Based Reasoning for Drug-Target Interaction Prediction. arXiv.
  3. Ebrahimi, S., and Asudeh, A. (2026, April 23). A Survey on Reliability, Transparency, Accountability, and Fairness in LLM-based Multi-Agent Systems through the Responsibility Lens. submitted. Available online: https://www.researchgate.net/profile/Abolfazl-Asudeh/publication/397650899_A_Survey_on_Reliability_Transparency_Accountability_and_Fairness_in_LLM-based_Multi-Agent_Systems_through_the_Responsibility_Lens/links/691933f9de8143098271909a/A-Survey-on-Reliability-Transparency-Accountability-and-Fairness-in-LLM-based-Multi-Agent-Systems-through-the-Responsibility-Lens.pdf.
  4. Afolabi, H., Afolabi, Z., Friel, E., Roberts, J., Ji-Xu, A., Chen, L., Ogbomo, E., Imevbore, E., Eneje, P., and El Ouahidi, W. (2025, January 6–7). Faithful or Just Plausible? Evaluating Faithfulness for Medical Reasoning in Closed-Source LLMs. Proceedings of the Second Workshop on GenAI for Health: Potential, Trust, and Policy Compliance, San Diego, CA, USA.
  5. Edwards, 2021, The EU AI Act: A summary of its significance and scope, Artif. Intell. (EU AI Act), 1, 25
  6. Garg, 2025, Explainable AI & Model Interpretability in Healthcare: Challenges & Future Directions, Eksplorium-Bul. Pus. Teknol. Bahan Galian Nukl., 46, 104
  7. Bauer, S.J. (2025). Responsible Artificial Intelligence in Healthcare: Impact on Patient Trust and Satisfaction with AI-Enabled Clinical Decision Support Systems. [Ph.D. Thesis, South College].

Cited by

See all 3

Metadata

Authors
Collaco, B. G., Prabha, S., Gomez-Cabello, C. A., Haider, S. A., Genovese, A., Wood, N. G., Gopala, N., Raman, R., Hester, E. O., & Forte, A. J.
Affiliations
  • Division of Plastic Surgery, Mayo Clinic, 4500 San Pablo Rd S, Jacksonville, FL 32224, USA
  • Mayo Clinic (US)
Corresponding author
Forte, Antonio Jorge
Venue
Bioengineering
Publisher
MDPI AG
Published
2026-04-28
Volume
13
Issue
5
Pages
513
ISSN
2306-5354 (ISSN-L)
OpenAlex
W7159842253
Cited by
3
References
40
Citations by year
2026: 3
FWCI
2.86
Type
journal article
Language
en
Open access
Locations
  • Bioengineering (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Schmidt Sciences
  • Richard M. Schulze Family Foundation
  • Gerstner Philanthropies
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
agentic AI, explainable AI, artificial intelligence in healthcare, multi-agent systems, post-hoc explanation, task decomposition, tool use, human-in-the-loop oversight, clinical decision support, radiology, neurology, biomedical research, transparency and accountability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review. https://app.cochat.ai/papers/10.3390/bioengineering13050513
The paper
Collaco, B. G., Prabha, S., Gomez-Cabello, C. A., Haider, S. A., Genovese, A., Wood, N. G., Gopala, N., Raman, R., Hester, E. O., & Forte, A. J. (2026). Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review. Bioengineering. https://doi.org/10.3390/bioengineering13050513

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.