From Blackbox to Explainable AI in Healthcare: Existing Tools and Case Studies
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Introduction. Artificial intelligence (AI) models have been employed to automate decision-making, from commerce to more critical fields directly affecting human lives, including healthcare. Although the vast majority of these proposed AI systems are considered black box models that lack explainability, there is an increasing trend of attempting to create medical explainable Artificial Intelligence (XAI) systems using approaches such as attention mechanisms and surrogate models. An AI system is said to be explainable if humans can tell how the system reached its decision. Various XAI-driven healthcare approaches and their performances in the current study are discussed. The toolkits used in local and global post hoc explainability and the multiple techniques for explainability pertaining the Rational, Data, and Performance explainability are discussed in the current study. Methods. The explainability of the artificial intelligence model in the healthcare domain is implemented through the Local Interpretable Model-Agnostic Explanations and Shapley Additive Explanations for better comprehensibility of the internal working mechanism of the original AI models and the correlation among the feature set that influences decision of the model. Results. The current state-of-the-art XAI-based and future technologies through XAI are reported on research findings in various implementation aspects, including research challenges and limitations of existing models. The role of XAI in the healthcare domain ranging from the earlier prediction of future illness to the disease’s smart diagnosis is discussed. The metrics considered in evaluating the model’s explainability are presented, along with various explainability tools. Three case studies about the role of XAI in the healthcare domain with their performances are incorporated for better comprehensibility. Conclusion. The future perspective of XAI in healthcare will assist in obtaining research insight in the healthcare domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Kumar, Y., Koul, A., Singla, R., & Ijaz, M. F. (2023) · Journal of Ambient Intelligence and Humanized Computing · doi:10.1007/s12652-021-03612-z · Cited by 1,040
RetractedArrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · cited 1×
“Table 3 represents the various characteristics like the acceptable input type, a local and global explainable toolkit feature, and the model-specific and agnostic feature associated with the toolkits [38].”
BackgroundExplainability for artificial intelligence in healthcare: a multidisciplinary perspective
the Precise4Q consortium, Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01332-6 · Cited by 2,203 · cited 1×
“Explaining Articial Intelligence (XAI) [1, 2] is a eld of machine intelligence engineering that ensures that complicated technics are made accessible and customizable for eective decision-making in the eld of science and technology.”
BackgroundDefinitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 1×
“In these contexts, explainable approaches must contend that individual features lack contextual significance, rendering the task more difficult than on datasets with more meaningful features [74].”
BackgroundCausability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885 · cited 1×
“*e current models in healthcare and medicine have limitations in grappling with real-world complexities and uncertainties [7].”
BackgroundPotential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · cited 1×
“Also, some of the predictive models are biased and tend to produce misleading outcomes [6]. challenges of the adaptability, transparency, and biased behavior of the existing models have provoked transparency in the design models in conventional Artificial Intelligence. goal is to assist the…”
BackgroundLee, D., & Yoon, S. N. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18010271 · Cited by 1,017 · cited 1×
“Severalof the most efficient machine learning systems’ properties seem suspicious in making the decisions [57].”
BackgroundFooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900
Artificial Intelligence and Black‐Box Medical Decisions: Accuracy versus Explainability
London, A. J. (2019) · Hastings Center Report · doi:10.1002/hast.973 · Cited by 801
Explainable Deep Learning Models in Medical Image Analysis
Singh, A., Sengupta, S., & Lakshminarayanan, V. (2020) · Journal of Imaging · doi:10.3390/jimaging6060052 · Cited by 718 · cited 1×
“*is field benefits AI for various activities, including decision making, predictive evaluation, analysis, risk evaluation, and regulation [21].”
BackgroundModel-Agnostic Interpretability of Machine Learning
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05386 · Cited by 699 · cited 1×
“*e same model can be applied to various classification and prediction models [29].”
MethodAntoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629
Evaluating the quality of machine learning explanations: a survey on methods and metrics
Zhou, J., Gandomi, A. H., Chen, F., & Holzinger, A. (2021) · Electronics · doi:10.3390/electronics10050593 · Cited by 606 · cited 1×
“*e toolkits are efficient in evaluating the explainability and understandability of the model through various evaluation metrics [39], which are categorized as Rationale explanation, Data explanation, and Safety and performance explanation [40].”
BackgroundInterpretable Machine Learning in Healthcare
Ahmad, M. A., Eckert, C., & Teredesai, A. (2018) · Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics · doi:10.1145/3233547.3233667 · Cited by 583 · cited 1×
“Bringing simplicity to ML models by including precise information on why the model adopted a particular action has been one of XAI’s goals, developing further understandable and transparent ML models while retaining high output standards is another [8].”
MethodLoyola-Gonzalez, O. (2019) · IEEE Access · doi:10.1109/access.2019.2949286 · Cited by 559
Metrics for explainable AI: challenges and prospects
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018) · arXiv · doi:10.48550/arxiv.1812.04608 · Cited by 518
Lauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., & Thiesson, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17431-x · Cited by 448
El-Sappagh, S., Alonso, J. M., Islam, S. M. R., Sultan, A. M., & Kwak, K. S. (2021) · Scientific Reports · doi:10.1038/s41598-021-82098-3 · Cited by 363
Magesh, P. R., Myloth, R. D., & Tom, R. J. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104041 · Cited by 295
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2019) · arXiv · doi:10.48550/arxiv.1909.03012 · Cited by 268 · cited 1×
“[37] have proposed a toolkit AI Explainability 360, which is available as open-access software with eight state-of-the-art features in handling the design and execution of the explainable model that includes Boolean Decision Rules via Column Generation (BRCG), Generalized Linear Rule Models (GLRM), ProtoDash, ProfWeig…”
MethodLakhan, A., Mohammed, M. A., Nedoma, J., Martinek, R., Tiwari, P., Vidyarthi, A., Alkhayyat, A., & Wang, W. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3165945 · Cited by 224
An Empirical Study of Model-Agnostic Techniques for Defect Prediction Models
Jiarpakdee, J., Tantithamthavorn, C. K., Dam, H. K., & Grundy, J. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2982385 · Cited by 148
Explainable AI and Multi-Modal Causability in Medicine
Holzinger, A. (2021) · i-com · doi:10.1515/icom-2020-0024 · Cited by 87 · cited 1×
“Explaining Artificial Intelligence (XAI) [1, 2] is a field of machine intelligence engineering that ensures that complicated technics are made accessible and customizable for effective decision-making in the field of science and technology.”
BackgroundExplainability and Interpretability: Keys to Deep Medicine
Shaban-Nejad, A., Michalowski, M., & Buckeridge, D. L. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-53352-6_1 · Cited by 35
Cited by
Shaping the future of healthcare: ethical clinical challenges and pathways to trustworthy AI
Goktas, P., & Grzybowski, A. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14051605 · Cited by 336
“In this process, internal organizational efforts bolster reliability and fairness while managing potentially harmful external influences such as regulatory gaps or privacy risks.”
Large Language Models in Healthcare and Medical Applications: A Review
Maity, S., & Saikia, M. J. (2025) · Bioengineering · doi:10.3390/bioengineering12060631 · Cited by 249
“As LLMs become more integrated into clinical workflows, ongoing research and development in explainability will be essential for responsible and effective deployment [78–81].”
Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond
Nasir, S., Khan, R. A., & Bai, S. (2024) · IEEE Access · doi:10.1109/access.2024.3369912 · Cited by 152
Moazemi, S., Vahdati, S., Li, J., Kalkhoff, S., Castano, L. J. V., Dewitz, B., Bibo, R., Sabouniaghdam, P., Tootooni, M. S., Bundschuh, R. A., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1109411 · Cited by 97
Shifa, N., Saleh, M., Akbari, Y., & Al Maadeed, S. (2025) · Clinical Imaging · doi:10.1016/j.clinimag.2025.110492 · Cited by 45
Metadata
- Authors
- Srinivasu, P. N., Sandhya, N., Jhaveri, R. H., & Raut, R.
- Venue
- Mobile Information Systems
- Publisher
- Wiley
- Published
- 2022-06-13
- Volume
- 2022
- Pages
- 1-20
- ISSN
- 1574-017X (ISSN-L)
- 1875-905X
- OpenAlex
- W4282821541
- Cited by
- 171
- References
- 27
- FWCI
- 11.47
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.