Essential properties and explanation effectiveness of explainable artificial intelligence in healthcare: A systematic review
Open accessAbstract
Background: Significant advancements in the field of information technology have influenced the creation of trustworthy explainable artificial intelligence (XAI) in healthcare. Despite improved performance of XAI, XAI techniques have not yet been integrated into real-time patient care. Objective: The aim of this systematic review is to understand the trends and gaps in research on XAI through an assessment of the essential properties of XAI and an evaluation of explanation effectiveness in the healthcare field. Methods: A search of PubMed and Embase databases for relevant peer-reviewed articles on development of an XAI model using clinical data and evaluating explanation effectiveness published between January 1, 2011, and April 30, 2022, was conducted. All retrieved papers were screened independently by the two authors. Relevant papers were also reviewed for identification of the essential properties of XAI (e.g., stakeholders and objectives of XAI, quality of personalized explanations) and the measures of explanation effectiveness (e.g., mental model, user satisfaction, trust assessment, task performance, and correctability). Results: Six out of 882 articles met the criteria for eligibility. Artificial Intelligence (AI) users were the most frequently described stakeholders. XAI served various purposes, including evaluation, justification, improvement, and learning from AI. Evaluation of the quality of personalized explanations was based on fidelity, explanatory power, interpretability, and plausibility. User satisfaction was the most frequently used measure of explanation effectiveness, followed by trust assessment, correctability, and task performance. The methods of assessing these measures also varied. Conclusion: XAI research should address the lack of a comprehensive and agreed-upon framework for explaining XAI and standardized approaches for evaluating the effectiveness of the explanation that XAI provides to diverse AI stakeholders.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Within the GDPR of the European Union, transparency is a fundamental principle for data processing.
“Within the General Data Protection Regulation (GDPR) of the European Union, transparency is a fundamental principle for data processing [15].”
From Explainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare (2023) · cites this paper as [15]
Privacy laws such as GDPR, HIPAA and PIPL all include clauses that state that the handling of healthcare data should be transparent, which means that AI algorithms that work with these data should be transparent and explainable as well.
“Privacy laws such as GDPR, HIPAA and PIPL all include clauses that state that the handling of healthcare data should be transparent, which means that AI algorithms that work with these data should be transparent and explainable as well.”
From Explainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare (2023) · cites this paper as [15]
Consistent with prior findings, only a small number of studies formally evaluated the effectiveness of XAI in clinical settings.
“Consistent with prior findings [44], only a small number of studies formally evaluated the effectiveness of XAI in clinical settings.”
From Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review (2026) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The PRISMA 2020 statement: an updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01626-4 · Cited by 13,874 · see citing works
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423 · see citing works
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · see citing works
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · PLoS Medicine · doi:10.1371/journal.pmed.1003583 · Cited by 5,148 · see citing works
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · see citing works
Explainability for artificial intelligence in healthcare: a multidisciplinary perspective
the Precise4Q consortium, Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01332-6 · Cited by 2,203 · see citing works
Shin, D. (2021) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2020.102551 · Cited by 1,548 · see citing works
DARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360 · see citing works
Artificial intelligence in cardiology
Johnson, K. W., Torres Soto, J., Glicksberg, B. S., Shameer, K., Miotto, R., Ali, M., Ashley, E., & Dudley, J. T. (2018) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2018.03.521 · Cited by 1,202 · see citing works
A governance model for the application of AI in health care
Reddy, S., Allan, S., Coghlan, S., & Cooper, P. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz192 · Cited by 979 · see citing works
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811 · see citing works
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785 · see citing works
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717 · see citing works
Langer, M., Oster, D., Speith, T., Hermanns, H., Kästner, L., Schmidt, E., Sesing, A., & Baum, K. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103473 · Cited by 652 · see citing works
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598 · cited 1× · see citing works
“Thus, adding the objective of explainability to control AI can prevent erroneous outcomes and enable debugging, which reduces the risk of patient harm [24].”
BackgroundDARPA's explainable artificial intelligence (XAI) program
Gunning, D. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3308446 · Cited by 546 · see citing works
Explainable Artificial Intelligence: Objectives, Stakeholders, and Future Research Opportunities
Meske, C., Bunde, E., Schneider, J., & Gersch, M. (2020) · Information Systems Management · doi:10.1080/10580530.2020.1849465 · Cited by 426 · see citing works
The practice of medical technology
Timmermans, S., & Berg, M. (2003) · Sociology of Health & Illness · doi:10.1111/1467-9566.00342 · Cited by 407 · see citing works
The EU Approach to Ethics Guidelines for Trustworthy Artificial Intelligence
Smuha, N. A. (2019) · Computer Law Review International · doi:10.9785/cri-2019-200402 · Cited by 356 · see citing works
Payrovnaziri, S. N., Chen, Z., Rengifo‐Moreno, P., Miller, T., Bian, J., Chen, J. H., Liu, X., & He, Z. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa053 · Cited by 318 · see citing works
Chou, Y.-L., Moreira, C., Bruza, P., Ouyang, C., & Jorge, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.003 · Cited by 302 · see citing works
Čartolovni, A., Tomičić, A., & Lazić Mosler, E. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104738 · Cited by 267 · see citing works
Meaningful Explanations of Black Box AI Decision Systems
Pedreschi, D., Giannotti, F., Guidotti, R., Monreale, A., Ruggieri, S., & Turini, F. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019780 · Cited by 233 · see citing works
Holzinger, A., Dehmer, M., Emmert‐Streib, F., Cucchiara, R., Augenstein, I., Ser, J. D., Samek, W., Jurišica, I., & Díaz-Rodríguez, N. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.10.007 · Cited by 215 · see citing works
Kenny, E. M., Ford, C., Quinn, M., & Keane, M. T. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103459 · Cited by 211 · see citing works
Accelerating Detection of Lung Pathologies with Explainable Ultrasound Image Analysis
Born, J., Wiedemann, N., Cossio, M., Buhre, C., Brändle, G., Leidermann, K., Goulet, J., Aujayeb, A., Moor, M., Rieck, B., et al. (2021) · Applied Sciences · doi:10.3390/app11020672 · Cited by 166 · see citing works
On the Governance of Artificial Intelligence through Ethics Guidelines
LARSSON, S. (2020) · Asian Journal of Law and Society · doi:10.1017/als.2020.19 · Cited by 156 · see citing works
Explainable AI for Healthcare: From Black Box to Interpretable Models
Adadi, A., & Berrada, M. (2020) · Advances in Intelligent Systems and Computing · doi:10.1007/978-981-15-0947-6_31 · Cited by 137 · see citing works
FairSight: Visual Analytics for Fairness in Decision Making
Ahn, Y., & Lin, Y.-R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934262 · Cited by 126 · see citing works
Interpretable heartbeat classification using local model-agnostic explanations on ECGs
Neves, I., Folgado, D., Santos, S., Barandas, M., Campagner, A., Ronzio, L., Cabitza, F., & Gamboa, H. F. S. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104393 · Cited by 118 · see citing works
The Next Frontier: AI We Can Really Trust
Holzinger, A. (2021) · PKDD/ECML Workshops · doi:10.1007/978-3-030-93736-2_33 · Cited by 114 · see citing works
Rundo, L., Pirrone, R., Vitabile, S., Sala, E., & Gambino, O. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103479 · Cited by 113 · see citing works
Challenges of Explaining the Behavior of Black-Box AI Systems
Asatiani, A., Malo, P., Nagbøl, P. R., et al. (2020) · MIS Quarterly Executive · doi:10.17705/2msqe.00037 · Cited by 107 · see citing works
Wulf, A. J., & Seizov, O. (2022) · AI & Society · doi:10.1007/s00146-022-01424-z · Cited by 101 · see citing works
Sabol, P., Sinčák, P., Hartono, P., Kočan, P., Benetinová, Z., Blichárová, A., Verbóová, Ľ., Štammová, E., Antónia, S. F., & Anna, J. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103523 · Cited by 89 · see citing works
Veale, M. (2020) · European Journal of Risk Regulation · doi:10.1017/err.2019.65 · Cited by 85 · see citing works
Explainable AI, But Explainable to Whom? An Exploratory Case Study of xAI in Healthcare
Gerlings, J., Jensen, M. S., & Shollo, A. (2021) · Intelligent systems reference library · doi:10.1007/978-3-030-83620-7_7 · Cited by 85 · see citing works
Artificial Intelligence and Clinical Decision Making: The New Nature of Medical Uncertainty
Harish, V., Morgado, F., Stern, A. D., & Das, S. (2020) · Academic Medicine · doi:10.1097/acm.0000000000003707 · Cited by 73 · see citing works
Bove, C., Aigrain, J., Lesot, M., Tijus, C., & Detyniecki, M. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511139 · Cited by 68 · see citing works
Mental Models of Mere Mortals with Explanations of Reinforcement Learning
Anderson, A. W., Dodge, J. E., Sadarangani, A., Juozapaitis, Z., Newman, E., Irvine, J., Chattopadhyay, S., Olson, M., Fern, A., & Burnett, M. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3366485 · Cited by 63 · see citing works
XAI for myo-controlled prosthesis: Explaining EMG data for hand gesture classification
Gozzi, N., Malandri, L., Mercorio, F., & Pedrocchi, A. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.108053 · Cited by 63 · see citing works
Personalized explanation in machine learning: A conceptualizationPreprint
Schneider, J., & Handali, J. P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.00770 · Cited by 56 · see citing works
Interpretable and Lightweight 3-D Deep Learning Model for Automated ACL Diagnosis
Jeon, Y., Yoshino, K., Hagiwara, S., Watanabe, A., Quek, S. T., Yoshioka, H., & Feng, M. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3081355 · Cited by 54 · see citing works
Palladino, N. (2021) · Telecommunications Policy · doi:10.1016/j.telpol.2021.102149 · Cited by 48 · see citing works
A Comprehensive Explanation Framework for Biomedical Time Series Classification
Ivaturi, P., Gadaleta, M., Pandey, A. C., Pazzani, M. J., Steinhubl, S. R., & Quer, G. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3060997 · Cited by 46 · see citing works
Explainable Artificial Intelligence for Bias Detection in COVID CT-Scan Classifiers
Palatnik de Sousa, I., Vellasco, M. M. B. R., & Costa da Silva, E. (2021) · Sensors · doi:10.3390/s21165657 · Cited by 43 · see citing works
Faster R-CNN with Inception V2 for Fingertip Detection in Homogenous Background Image
Alamsyah, D., & Fachrurrozi, M. (2019) · Journal of Physics Conference Series · doi:10.1088/1742-6596/1196/1/012017 · Cited by 41 · see citing works
Tan, W., Guan, P., Wu, L., Chen, H., Li, J., Ling, Y., Fan, T., Wang, Y., Li, J., & Yan, B. (2021) · Annals of Translational Medicine · doi:10.21037/atm-21-1171 · Cited by 25 · see citing works
Souza, J., & Leung, C. K.-S. (2021) · Springer eBooks · doi:10.1007/978-3-030-76409-8_4 · Cited by 18 · see citing works
Wulf, A. J., & Seizov, O. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4478311 · Cited by 15 · see citing works
Alahmadi, A., Davies, A., Royle, J., Goodwin, L., Cresswell, K., Arain, Z., Vigo, M., & Jay, C. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104281 · Cited by 14 · see citing works
Mitra, A., Mondal, J., & Das, A. (2021) · Chapman & Hall/CRC eBooks · doi:10.1201/9780429318078 · Cited by 11 · see citing works
Derathé, A., Reche, F., Jannin, P., et al. (2021) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-021-02422-0 · Cited by 9 · see citing works
Basic Issues and Challenges on Explainable Artificial Intelligence (XAI) in Healthcare Systems
Oladipo, I. D., Awotunde, J. B., AbdulRaheem, M., & Salihu, S. A. (2022) · Advances in medical technologies and clinical practice book series · doi:10.4018/978-1-6684-3791-9.ch011 · Cited by 9 · see citing works
Ethical, Societal and Legal Issues in Deep Learning for Healthcare
Panigutti, C., Monreale, A., Comandé, G., et al. (2022) · WORLD SCIENTIFIC (EUROPE) eBooks · doi:10.1142/9781800610941_0009 · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [2]Kocielnik R. Amershi S. Bennett P.N. Will you accept an imperfect AI? Exploring designs for adjusting end-user expectations of AI systems, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems J. ACM 2019 1 14 10.1145/3290605.3300641 · DOI · Paper page
Cited by
Sorted by citations
A Review of Explainable Artificial Intelligence in Healthcare
Sadeghi, Z., Alizadehsani, R., CIFCI, M. A., Kausar, S., Rehman, R., Mahanta, P., Bora, P. K., Almasri, A., Alkhawaldeh, R. S., Hussain, S., et al. (2024) · Computers and Electrical Engineering · doi:10.1016/j.compeleceng.2024.109370 · Cited by 598 · see citing works
Explainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare
Hulsen, T. (2023) · AI · doi:10.3390/ai4030034 · Cited by 305 · see citing works
Rosenbacke, R., Melhus, Å., McKee, M., & Stuckler, D. (2024) · JMIR AI · doi:10.2196/53207 · Cited by 305 · see citing works
A literature review on applications of explainable artificial intelligence (XAI)
Kalasampath, K., Spoorthi, K. N., Sajeev, S., Kuppa, S. S., Ajay, K., & Maruthamuthu, A. (2025) · IEEE Access · doi:10.1109/access.2025.3546681 · Cited by 180 · see citing works
Human-centered evaluation of explainable AI applications: a systematic review
Kim, J., Maathuis, H., & Sent, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1456486 · Cited by 130 · see citing works
Rony, M. K. K., Numan, S. M., Akter, K., Tushar, H., Debnath, M., Johra, F. T., Akter, F., Mondal, S., Das, M., Uddin, M. J., et al. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e36702 · Cited by 93 · see citing works
Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review
Ebraheem, M., Toghranegar, J., Bridge2AI-Voice Consortium, Bensoussan, Y., & Templeton, J. M. (2026) · Journal of Medical Internet Research · doi:10.2196/83790
- Authors
- Jung, J., Lee, H., Jung, H., & Kim, H.
- Affiliations
- Seoul National University (KR)
- University of Seoul (KR)
- Corresponding author
- Kim, Hyeoneui
- Venue
- Heliyon
- Publisher
- Elsevier BV
- Published
- 2023-05
- Volume
- 9
- Issue
- 5
- Pages
- e16110
- ISSN
- 2405-8440 (ISSN-L)
- PMC
- PMC10205582
- OpenAlex
- W4378515194
- Cited by
- 87
- References
- 56
- Citations by year
- 2026: 31 · 2025: 27 · 2024: 25 · 2023: 4
- FWCI
- 1.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Heliyon (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Seoul National University Open Repository (Seoul National University) (submittedVersion)
- Funders
- Ministry of Education
- National Research Foundation of Korea
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- explainable AI, health care, personalized explanations, user satisfaction, trust evaluation, correctability, task performance, mental models, clinical data, interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
