The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Application of explainable artificial intelligence in medical health: A systematic review of interpretability methods
Open accessAbstract
This paper investigates the applications of explainable AI (XAI) in healthcare, which aims to provide transparency, fairness, accuracy, generality, and comprehensibility to the results obtained from AI and ML algorithms in decision-making systems. The black box nature of AI and ML systems has remained a challenge in healthcare, and interpretable AI and ML techniques can potentially address this issue. Here we critically review previous studies related to the interpretability of ML and AI methods in medical systems. Descriptions of various types of XAI methods such as layer-wise relevance propagation (LRP), Uniform Manifold Approximation and Projection (UMAP), Local Interpretable Model-agnostic Explanations (LIME), SHapley Additive exPlanations (SHAP), ANCHOR, contextual importance and utility (CIU), Training calibration-based explainers (TraCE), Gradient-weighted Class Activation Mapping (Grad-CAM), t-distributed Stochastic Neighbor Embedding (t-SNE), NeuroXAI, Explainable Cumulative Fuzzy Class Membership Criterion (X-CFCMC) along with the diseases which can be explained through these methods are provided throughout the paper. The paper also discusses how AI and ML technologies can transform healthcare services. The usability and reliability of the presented methods are summarized, including studies on the usability and reliability of XGBoost for mediastinal cysts and tumors, a 3D brain tumor segmentation network, and the TraCE method for medical image analysis. Overall, this paper aims to contribute to the growing field of XAI in healthcare and provide insights for researchers, practitioners, and decision-makers in the healthcare industry. Finally, we discuss the performance of XAI methods applied in medical health care systems. It is also needed to mention that a brief implemented method is provided in the methodology section.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2008) · Journal of Machine Learning Research · Cited by 33,051
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Dwivedi, Y. K., Hughes, L., Ismagilova, E., Aarts, G., Coombs, C., Crick, T., Duan, Y., Dwivedi, R., Edwards, J., Eirug, A., et al. (2021) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2019.08.002 · Cited by 4,611
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524
Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks
Chattopadhay, A., Sarkar, A., Howlader, P., & Balasubramanian, V. N. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00097 · Cited by 3,639
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162
Shin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548
A governance model for the application of AI in health care
Reddy, S., Allan, S., Coghlan, S., & Cooper, P. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz192 · Cited by 979
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Effect of Data Scaling Methods on Machine Learning Algorithms and Model Performance
Ahsan, M., Mahmud, M., Saha, P., Gupta, K., & Siddique, Z. (2021) · Technologies · doi:10.3390/technologies9030052 · Cited by 744
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
DARPA's explainable artificial intelligence (XAI) program
Gunning, D. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3308446 · Cited by 546
Lauritsen, S. M., Kristensen, M., Olsen, M. V., Larsen, M. S., Lauritsen, K. M., Jørgensen, M. J., Lange, J., & Thiesson, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17431-x · Cited by 448
2018 IEEE Winter Conference on Applications of Computer Vision (WACV)
(2018) · doi:10.1109/wacv41190.2018 · Cited by 350
Grad-CAM: Why did you say that?
Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., & Batra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07450 · Cited by 309
Magesh, P. R., Myloth, R. D., & Tom, R. J. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104041 · Cited by 295
Chen, J., Hu, C., Chen, L., Tang, L., Zhu, Y., Xu, X., Chen, L., Gao, H., Lu, X., Yu, L., et al. (2020) · Engineering · doi:10.1016/j.eng.2020.02.006 · Cited by 242
Explainable COVID-19 Detection Using Chest CT Scans and Deep Learning
Alshazly, H., Linse, C., Barth, E., & Martinetz, T. (2021) · Sensors · doi:10.3390/s21020455 · Cited by 216
Holzinger, A., Dehmer, M., Emmert-Streib, F., Cucchiara, R., Augenstein, I., Ser, J. D., Samek, W., Jurisica, I., & Díaz-Rodríguez, N. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.10.007 · Cited by 213
Explanations in knowledge systems: design for explainable expert systems
Swartout, W., Paris, C., & Moore, J. (1991) · IEEE Expert · doi:10.1109/64.87686 · Cited by 200
Explainability of deep neural networks for MRI analysis of brain tumors
Zeineldin, R. A., Karar, M. E., Elshaer, Z., Coburger, J., Wirtz, C. R., Burgert, O., & Mathis-Ullrich, F. (2022) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-022-02619-x · Cited by 191
ElShawi, R., Sherif, Y., Al‐Mallah, M., & Sakr, S. (2021) · Computational Intelligence · doi:10.1111/coin.12410 · Cited by 177
Explainable Artificial Intelligence for Human Decision Support System in the Medical Domain
Knapič, S., Malhi, A., Saluja, R., & Främling, K. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3030037 · Cited by 171
Ahsan, M. M., Gupta, K. D., Islam, M. M., Sen, S., Rahman, M. L., & Hossain, M. S. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make2040027 · Cited by 162
COVID-Transformer: Interpretable COVID-19 Detection Using Vision Transformer for Healthcare
Shome, D., Kar, T., Mohanty, S. N., Tiwari, P., Muhammad, K., AlTameem, A., Zhang, Y., & Saudagar, A. K. J. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph182111086 · Cited by 154
Rajpal, S., Lakhyani, N., Singh, A., Kohli, R., & Kumar, N., V (2021) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2021.110749 · Cited by 148
Kamal, M. S., Northcote, A., Chowdhury, L., Dey, N., Crespo, R. A. G., & Herrera‐Viedma, E. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3107056 · Cited by 145
Building Explainable Artificial Intelligence Systems
Core, M. G., Lane, H. C., van Lent, M., Gomboc, D., Solomon, S., & Milton Rosenberg, B. (2006) · Defense Technical Information Center (DTIC) · doi:10.21236/ada459166 · Cited by 141
Chest Radiographs in Congestive Heart Failure: Visualizing Neural Network Learning
Seah, J. C. Y., Tang, J. S. N., Kitchen, A., Gaillard, F., & Dixon, A. F. R. (2018) · Radiology · doi:10.1148/radiol.2018180887 · Cited by 135
Zhang, Y., Zhang, Z., Zhang, X., & Wang, S. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.06.021 · Cited by 120
Sabol, P., Sinčák, P., Hartono, P., Kočan, P., Benetinová, Z., Blichárová, A., Verbóová, Ľ., Štammová, E., Antónia, S. F., & Anna, J. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103523 · Cited by 106
Toward Human–AI Interfaces to Support Explainability and Causability in Medical AI
Holzinger, A., & Müller, H. (2021) · Computer · doi:10.1109/mc.2021.3092610 · Cited by 104
Visual interpretability in 3D brain tumor segmentation network
Saleem, H. A., Shahid, A. R., & Raza, B. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104410 · Cited by 103
A Comprehensive Unsupervised Framework for Chronic Kidney Disease Prediction
Antony, L., Azam, S., Ignatious, E., Quadir, R., Beeravolu, A. R., Jonkman, M., & Boer, F. D. (2021) · IEEE Access · doi:10.1109/access.2021.3109168 · Cited by 98
Windisch, P. Y., Weber, P., Fürweger, C., Ehret, F., Kufeld, M., Zwahlen, D. R., & Muacevic, A. (2020) · Neuroradiology · doi:10.1007/s00234-020-02465-1 · Cited by 94
Ahmad, A., Akbar, S., Tahir, M., Hayat, M., & Ali, F. (2022) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2022.104516 · Cited by 90
Cabitza, F., Campagner, A., Malgieri, G., Natali, C., Schneeberger, D., Stoeger, K., & Holzinger, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118888 · Cited by 89
Umair, M., Khan, M. S., Ahmed, F., Baothman, F. A., Alqahtani, F., Alian, M., & Ahmad, J. E. (2021) · Sensors · doi:10.3390/s21175813 · Cited by 80
Gabbay, F., Bar‐Lev, S., Montano, O., & Hadad, N. (2021) · Applied Sciences · doi:10.3390/app112110417 · Cited by 78
Karim, M. R., Jiao, J., Döhmen, T., Cochez, M., Beyan, O., Rebholz‐Schuhmann, D., & Decker, S. (2021) · IEEE Access · doi:10.1109/access.2021.3062493 · Cited by 71
Thiagarajan, J. J., Thopalli, K., Rajan, D., & Turaga, P. K. (2022) · Scientific Reports · doi:10.1038/s41598-021-04529-5 · Cited by 67
Hussain, S. M., Buongiorno, D., Altini, N., Berloco, F., Prencipe, B., Moschetta, M., Bevilacqua, V., & Brunetti, A. (2022) · Applied Sciences · doi:10.3390/app12126230 · Cited by 63
Explainable Artificial Intelligence for Bias Detection in COVID CT-Scan Classifiers
Palatnik de Sousa, I., Vellasco, M. M. B. R., & Costa da Silva, E. (2021) · Sensors · doi:10.3390/s21165657 · Cited by 43
Kim, K. H., Koo, H., Lee, B., Yoon, S., & Sohn, M. (2021) · Journal of the Korean Physical Society · doi:10.1007/s40042-021-00202-2 · Cited by 43
An Artificial Intelligence Approach to Bloodstream Infections Prediction
Pai, K.-C., Wang, M.-S., Chen, Y., Tseng, C.-H., Liu, P., Chen, L.-C., Sheu, R., & Wu, C. (2021) · Journal of Clinical Medicine · doi:10.3390/jcm10132901 · Cited by 37
Ripoll, D. R., Chaudhury, S., & Wallqvist, A. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008864 · Cited by 32
Ghafouri‐Fard, S., Taheri, M. N., Omrani, M. D., Daaee, A., & Mohammad‐Rahimi, H. (2020) · Journal of Molecular Neuroscience · doi:10.1007/s12031-020-01514-x · Cited by 31
2020 Digital Image Computing: Techniques and Applications (DICTA)
(2020) · doi:10.1109/dicta51227.2020 · Cited by 30
Yang, R., Ke, F., Liu, H., Zhou, M., & Cao, H. (2021) · IEEE Access · doi:10.1109/access.2021.3069211 · Cited by 24
Alorf, A. A. (2021) · Algorithms · doi:10.3390/a14060183 · Cited by 17
Debiasing pipeline improves deep learning model generalization for X-ray based lung nodule detectionPreprint
Horry, M. J., Chakraborty, S., Pradhan, B., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.09563 · Cited by 7
Cited by
Longo, L., Brcic, M., Cabitza, F., Choi, J., Confalonieri, R., Ser, J. D., Guidotti, R., Hayashi, Y., Herrera, F., Holzinger, A., et al. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102301 · Cited by 626
A Review of Explainable Artificial Intelligence in Healthcare
Sadeghi, Z., Alizadehsani, R., CIFCI, M. A., Kausar, S., Rehman, R., Mahanta, P., Bora, P. K., Almasri, A., Alkhawaldeh, R. S., Hussain, S., et al. (2024) · Computers and Electrical Engineering · doi:10.1016/j.compeleceng.2024.109370 · Cited by 598
Fahim, Y. A., Hasani, I. W., Kabba, S., & Ragab, W. M. (2025) · European Journal of Medical Research · doi:10.1186/s40001-025-03196-w · Cited by 329
Nasarian, E., Alizadehsani, R., Acharya, U. R., & Tsui, K.-L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102412 · Cited by 207
Explainable and interpretable artificial intelligence in medicine: a systematic bibliometric review
Frasca, M., La Torre, D., Pravettoni, G., & Cutica, I. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00114-7 · Cited by 190
Metadata
- Authors
- S Band, S., Yarahmadi, A., Hsu, C.-C., Biyari, M., Sookhak, M., Ameri, R., Dehzangi, I., Chronopoulos, A. T., & Liang, H.-W.
- Venue
- Informatics in Medicine Unlocked
- Publisher
- Elsevier BV
- Published
- 2023
- Volume
- 40
- Pages
- 101286
- ISSN
- 2352-9148 (ISSN-L)
- OpenAlex
- W4380201339
- Cited by
- 297
- References
- 54
- FWCI
- 8.18
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.