Unveiling Explainable AI in Healthcare: Current Trends, Challenges, and Future Directions
Open accessAbstract
This overview investigates the evolution and current landscape of eXplainable Artificial Intelligence (XAI) in healthcare, highlighting its implications for researchers, technology developers, and policymakers. Following the PRISMA protocol, we analysed 89 publications from January 2000 to June 2024, spanning 19 medical domains, with a focus on Neurology and Cancer as the most studied areas. Various data types are reviewed, including tabular data, medical imaging, and clinical text, offering a comprehensive perspective on XAI applications. Key findings identify significant gaps, such as the limited availability of public datasets, suboptimal data preprocessing techniques, insufficient feature selection and engineering, and the limited utilisation of multiple XAI methods. Additionally, the lack of standardised XAI evaluation metrics and practical obstacles in integrating XAI systems into clinical workflows are emphasised. We provide actionable recommendations, including the design of explainability-centric models, the application of diverse and multiple XAI methods, and the fostering of interdisciplinary collaboration. These strategies aim to guide researchers in building robust AI models, assist technology developers in creating intuitive and user-friendly AI tools, and inform policymakers in establishing effective regulations. Addressing these gaps will promote the development of transparent, reliable, and user-centred AI systems in healthcare, ultimately improving decision-…
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing evaluation practices for XAI in CDSS lack robustness and user-centered perspectives.
“Further reviews focusing on XAI in CDSS (e.g., Kim et al. [54] and Aziz et al. [55]) provide broad overviews of applied methods and clinical domains, but they consistently critique existing evaluation practices for lacking robustness and user-centered perspectives.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [55]
Gaps in XAI-CDSS include the need for better healthcare data handling, more comprehensive XAI evaluation methods tailored to usability, and stronger interdisciplinary collaboration to bridge the technical-clinical divide.
“Aziz et al. [55], analyzing studies up to 2024, identified gaps including the need for better healthcare data handling, more comprehensive XAI evaluation methods tailored to usability, and stronger interdisciplinary collaboration to bridge the technical-clinical divide.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [55]
Systematic reviews such as Aziz et al. urge the development of comprehensive XAI evaluation strategies that better assess clinical usability and impact.
“Systematic reviews such as Aziz et al. [55] explicitly urge the development of comprehensive XAI evaluation strategies that better assess clinical usability and impact.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [55]
Explainable AI in healthcare has been surveyed extensively, yet no existing review addresses the challenges posed by decoder-based LLMs.
“Explainable AI in healthcare has been surveyed extensively, yet no existing review addresses the challenges posed by decoder-based LLMs.”
From Explainability of decoder-only clinical large language models: A scoping review (2026) · cites this paper as [44]
Interpretability is increasingly recognized as essential for the clinical integration of AI.
“While interpretability is increasingly recognized as essential for the clinical integration of AI, further prospective and real-world studies are needed to quantify its impact on patient outcomes and define implementation best practices429,430.”
From Artificial intelligence-driven multi-omics approaches in Alzheimer's disease: Progress, challenges, and future directions (2025) · cites this paper as [430]
Cites
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223 · see citing works
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · see citing works
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524 · see citing works
Loh, H. W., Ooi, C. P., Seoni, S., Barua, P. D., Molinari, F., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107161 · Cited by 734 · see citing works
What do we need to build explainable AI systems for the medical domain?
Holzinger, A., Biemann, C., Pattichis, C. S., & Kell, D. B. (2017) · arXiv · doi:10.48550/arxiv.1712.09923 · Cited by 636 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629 · see citing works
El-Sappagh, S., Alonso, J. M., Islam, S. M. R., Sultan, A. M., & Kwak, K. S. (2021) · Scientific Reports · doi:10.1038/s41598-021-82098-3 · Cited by 363 · cited 5× · see citing works
“…of the datasets relevant to each application area are presented in Table 8 and 9, Img (1) [23] Txt (4) [24] Prv (12) Tab (11) [25], [26], [27], [28], [29] , [30], [31], [22], [32], [21], [33] Img (1) [34] Cancer (13) Pub (7) Tab (4) [21], [35]–[37] Img (3) [38]–[40] Prv (6) Tab (3) [33], [41],…”
BackgroundMethodInfluentialZihni, E., Madai, V. I., Livne, M., Galinovic, I., Khalil, A. A., Fiebach, J. B., & Frey, D. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231166 · Cited by 187 · see citing works
Vasey, B., Ursprung, S., Beddoe, B., Taylor, E. H., Marlow, N., Bilbro, N., Watkinson, P., & McCulloch, P. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.1276 · Cited by 154 · see citing works
Du, Y., Rafferty, A. R., McAuliffe, F. M., Wei, L., & Mooney, C. (2022) · Scientific Reports · doi:10.1038/s41598-022-05112-2 · Cited by 128 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Ren, F., Wei, J., Chen, Q., Hu, M., Yu, L., Mi, J., Zhou, X., Qin, D., Wu, J., & Wu, A. (2025) · Acta Pharmaceutica Sinica B · doi:10.1016/j.apsb.2025.07.030 · Cited by 97 · see citing works
Salimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T. (2025) · Informatics · doi:10.3390/informatics12040119 · Cited by 39 · see citing works
Ozgun, M. C., Pei, J., Hindriks, K., Donatelli, L., Liu, Q., & Wang, J. (2025) · Proceedings of the 34th ACM International Conference on Information and Knowledge Management · doi:10.1145/3746252.3761164 · Cited by 7 · see citing works
Explainability of decoder-only clinical large language models: A scoping review
Mishra, N., Abu-Hanna, A., & Calixto, I. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103501
- Authors
- Aziz, N. A., Manzoor, A., Mazhar Qureshi, M. D., Qureshi, M. A., & Rashwan, W.
- Affiliations
- Technological University Dublin (IE)
- Science Foundation Ireland (IE)
- Corresponding author
- Qureshi, M. Atif
- Venue
- medRxiv
- Publisher
- openRxiv
- Published
- 2024-08-10
- OpenAlex
- W4401477265
- Cited by
- 30
- References
- 11
- Citations by year
- 2026: 10 · 2025: 20
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- medRxiv (acceptedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- explainable AI, artificial intelligence in healthcare, XAI evaluation metrics, clinical decision support, diagnostic imaging, clinical text, feature selection, data preprocessing, neurology, cancer, AI regulation, AI transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
