CoChat
Sign Up
Cited by 305Explainable Artificial Intelligence (XAI)Artificial Intelligence in Healthcare and EducationMachine Learning in Healthcare

Explainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare

Open access
Hulsen, Tim·2023·AI
DOI 10.3390/ai4030034

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial Intelligence (AI) describes computer systems able to perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. Examples of AI techniques are machine learning, neural networks, and deep learning. AI can be applied in many different areas, such as econometrics, biometry, e-commerce, and the automotive industry. In recent years, AI has found its way into healthcare as well, helping doctors make better decisions (“clinical decision support”), localizing tumors in magnetic resonance images, reading and analyzing reports written by radiologists and pathologists, and much more. However, AI has one big risk: it can be perceived as a “black box”, limiting trust in its reliability, which is a very big issue in an area in which a decision can mean life or death. As a result, the term Explainable Artificial Intelligence (XAI) has been gaining momentum. XAI tries to ensure that AI algorithms (and the resulting decisions) can be understood by humans. In this narrative review, we will have a look at some central concepts in XAI, describe several challenges around XAI in healthcare, and discuss whether it can really help healthcare to advance, for example, by increasing understanding and trust. Finally, alternatives to increase trust in AI are discussed, as well as future research possibilities in the area of XAI.

Verified 2026-10-09 · 1 reference not checked yet

AI snapshot

Written from the paper’s full text.

TL;DR

This narrative review surveys the current status of Explainable Artificial Intelligence (XAI) in healthcare, covering the "black box" trust problem, legal/regulatory drivers, transparent versus post-hoc explainability, privacy and security trade-offs, and the tension between explainability and accuracy. The author argues that in healthcare accuracy may outweigh explainability, and that trust might better be built through robustness, fairness, reproducibility, and human–AI collaboration than through explanation alone.

Key findings

  • XAI is growing rapidly in the literature: the first mention of "explainable artificial intelligence" in a PubMed title, abstract or keyword was in 2018, and it has since been mentioned 511 times, of which more than half (310) appeared in 2022 or the first months of 2023; Embase shows a similar trend (queries performed 2023-03-06).
  • Explainability and accuracy can trade off: making a model fully transparent (a "glass box") may require simplification that reduces the accuracy of a highly predictive model (a "crystal ball"). Van der Veer et al. concluded citizens may value explainability in healthcare less than in non-healthcare domains and less than professionals assume, especially weighed against accuracy.
  • XAI is double-edged for privacy and security: it can help detect errors and malicious behavior, but explanations may reveal sensitive information or enable manipulation via reverse engineering. Saifullah et al. recommend differential private federated learning plus perturbation-based XAI methods when both privacy and explainability are required.
  • Explanations do not automatically produce trust: Le Merrer and Trédan show remote explainability cannot prevent a service provider from lying about the true reasons for its decisions, while Druce et al. found a statistically significant increase in user trust and acceptance using a three-fold explanation.

Methods

  • Design: narrative review of XAI concepts, issues, and challenges in healthcare, with no systematic review protocol described.
  • Data: no new data were created or analyzed; the paper draws on cited literature and a bibliometric query of PubMed and Embase for the term "explainable artificial intelligence" in titles, abstracts and keywords, performed on 2023-03-06.
  • Analysis: thematic discussion organized into sections covering transparency, legal/regulatory compliance (GDPR, the EU AI Act, HIPAA, PIPL), transparent vs. post-hoc explainability, privacy/security, human–AI collaboration, trust, scientific XAI (sXAI), the explainability–accuracy balance, measurement of explainability, and future complexity.

Limitations

  • The paper states no formal limitations section; it is explicitly a narrative review, and the text notes that explainability is difficult to measure because "the quality of an explanation is somewhat subjective."
  • Evident from the text: the review is a non-systematic narrative synthesis by a single author, so selection of cited work is not governed by a documented search protocol.
  • Evident from the text: the version provided is a preprint (preprints.org, posted 7 March 2023) marked "NOT PEER-REVIEWED," and the author is an employee of Philips Research (declared under Acknowledgements).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/ai4030034

Cites

72 of 73 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 20 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The Embase database shows a similar trend of increasing XAI mentions.

    “The results for the Embase database show a similar trend.”
    Quote in the paperNot a substantial claim
  • Findingcitation

    The first mention of XAI in a PubMed title, abstract or keywords was in 2018, in a paper about machine learning in neuroscience.

    “Its first mention in a PubMed title, abstract or keywords was in 2018, in a paper about machine learning in neuroscience [11].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11] Vu, 2018, A Shared Vision for Machine Learning in Neuroscience, J. Neurosci. source

      Not checked in the free run.

  • Findingown data

    XAI is being mentioned more and more in scientific publications.

    “XAI is being mentioned more and more in scientific publications, as can be seen in figure 1.”
    Quote in the paperNot a substantial claim
  • Interpretationnone given

    XAI can help organizations comply with laws and regulations that require transparency and explainability in AI systems.

    “Another advantage of XAI is that it can help organizations comply with laws and regulations that require transparency and explainability in AI systems.”
    Quote in the paperNot a substantial claim
  • Interpretationown data

    The growing number of XAI mentions shows the growing importance of XAI in (bio)medicine and healthcare.

    “This shows the growing importance of XAI in (bio)medicine and healthcare.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 11
  • Classified as probably not a substantial claim. 16

  • May combine two claims.

    Findingown data

    Since 2018, XAI has been mentioned a total of 511 times in PubMed, of which more than half (310) were in papers from 2022 or the first months of 2023.

    “Since then, it has been mentioned a total of 511 times, of which more than half (310) in papers from 2022 or the first months of 2023.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Trust in AI can exist on two levels: trust in the model versus trust in the prediction.

    “This trust can exist on two levels: trust in the model versus trust in the prediction [14].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Tjoa, 2020, A survey on explainable artificial intelligence (xai): Toward medical xai, IEEE Trans. Neural Netw. Learn. Syst. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    A low opacity of the translucent glass box (or high transparency of the model) can lead to a better understanding of the model, which in turn could increase trust.

    “A low opacity of the translucent glass box (or high transparency of the model) can lead to a better understanding of the model, which in turn could increase trust.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: AI algorithms can be perceived as black boxes, and this limits trust in their reliability, which is a major issue in healthcare where decisions can be life-or-death.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Explainability might be considered of less importance in healthcare, compared to accuracy.

    “Therefore, explainability might be considered of less importance in healthcare, compared to accuracy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Post-hoc models need to be explained by resorting to diverse means to enhance their interpretability, such as text explanations, visual explanations, local explanations, explanations by example, explanations by simplification and feature relevance explanations techniques.

    “The post-hoc models need to explained by resorting to diverse means to enhance their interpretability, such as text explanations, visual explanations, local explanations, explanations by example, explanations by simplification and feature relevance explanations techniques.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Vale, 2022, Explainable artificial intelligence (XAI) post-hoc explainability methods: Risks and limitations in non-discrimination law, AI Ethics source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Within healthcare, trust is a fundamental issue because important decisions might be taken based on the output of the AI algorithm.

    “Within healthcare, trust is a fundamental issue because important decisions might be taken based on the output of the AI algorithm.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: AI algorithms can be perceived as black boxes, and this limits trust in their reliability, which is a major issue in healthcare where decisions can be life-or-death.

  • May combine two claims.

    Premisecitation

    The EU Artificial Intelligence Act distinguishes between non-high-risk and high-risk AI systems, imposing limited transparency obligations on non-high-risk systems and many restrictions on high-risk systems.

    “The EU is currently working on the Artificial Intelligence Act [17] which makes a distinction between non-high-risk and high-risk AI systems.”
    • Also stated in: § 3. Legal and regulatory compliance · plain
      “On non-high-risk systems only limited transparency obligations are imposed, while for high-risk systems many restrictions are imposed on quality, documentation, traceability, transparency, human oversight, accuracy and robustness.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] 2019, Black-box vs. white-box: Understanding their advantages and weaknesses from a practical point of view, IEEE Access source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Linear models and rule-based models are very transparent, but usually have a lower performance than deep learning algorithms.

    “Linear models and rule-based models are very transparent, but usually have a lower performance than deep learning algorithms (figure 5 of [39]).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39] Solayman, 2023, Automatic COVID-19 prediction using explainable machine learning techniques, Int. J. Cogn. Comput. Eng.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Combining explainability of algorithms with privacy-preserving methods such as federated learning might help address privacy and security implications of XAI.

    “Combining explainability of algorithms with privacy-preserving methods such as federated learning [27] might help.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27] 2021, Dissecting scientific explanation in AI (sXAI): A case for medicine and healthcare, Artif. Intell. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Bias could be introduced by having a training dataset with low diversity, or by subjective responses of clinicians to a questionnaire.

    “A bias could be introduced by having a training dataset with low diversity, or by subjective responses of clinicians to a questionnaire.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    End users just want to be sure that the predictions done by the algorithm are accurate, which can be proven by showing them correct predictions from the past.

    “End users just want to be sure that the predictions done by the algorithm are accurate, which can be proven by showing them correct predictions from the past.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Robustness of an algorithm in healthcare can be proven by presenting good results based on long-term use in different patient populations.

    “Robustness of an algorithm in healthcare can be proven by presenting good results based on long-term use in different patient populations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Increasing complexity might make it almost impossible for end users in healthcare (clinicians and patients) to understand and trust the algorithms.

    “This increasing complexity might make it almost impossible for end users in healthcare (clinicians as well as patients) to understand and trust the algorithms.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: AI algorithms can be perceived as black boxes, and this limits trust in their reliability, which is a major issue in healthcare where decisions can be life-or-death.

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Making AI explainable is a difficult task, and it will be even more difficult when the complexity of AI algorithms continues to increase.

    “However, making AI explainable is a difficult task, and it will be even more difficult when the complexity of AI algorithms will continue to increase.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    With algorithms getting more complex, it might also be more difficult to make them explainable.

    “With algorithms getting more complex, it might also be more difficult to make them explainable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationcitation

    Trust in algorithms could be raised by ensuring robustness and by encouraging fairness.

    “For example, trust in algorithms could also be raised by ensuring robustness and by encouraging fairness [50].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [50] Charmet, 2022, Explainable artificial intelligence for cybersecurity: A literature survey, Ann. Telecommun. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Citizens may value explainability of AI systems in healthcare less than in non-healthcare domains and less than often assumed by professionals, especially when weighed against system accuracy.

    “Van der Veer et al. [40] concluded that citizens may indeed value explainability of AI systems in healthcare less than in non-healthcare domains and less than often assumed by professionals, especially when weighed against system accuracy”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [40] Gao, 2022, A review of the application of deep learning in the detection of Alzheimer's disease, Int. J. Cogn. Comput. Eng.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Fairness of an AI algorithm is concurrent with bias minimization.

    “Fairness of an AI algorithm is concurrent with bias minimization.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Privacy laws such as GDPR, HIPAA and PIPL all include clauses that state that the handling of healthcare data should be transparent, which means that AI algorithms that work with these data should be transparent and explainable as well.

    “Privacy laws such as GDPR, HIPAA and PIPL all include clauses that state that the handling of healthcare data should be transparent, which means that AI algorithms that work with these data should be transparent and explainable as well.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [15] Jung, 2023, Essential properties and explanation effectiveness of explainable artificial intelligence in healthcare: A systematic review, Heliyon source

      Not checked in the free run.

    • cites [19] Lim, 2022, Explainable AI, but explainable to whom? An exploratory case study of xAI in healthcare, Handbook of Artificial Intelligence in Healthcare: Practicalities and Prospects

      Not checked in the free run.

    • cites [21] Phillips, P.J., Hahn, C.A., Fontana, P.C., Broniatowski, D.A., and Przybocki, M.A. (2020). Four Principles of Explainable Artificial Intelligence, National Institute of Standards and Technology. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Arrieta et al. classified studies on XAI into two approaches: transparent models and post-hoc models.

    “Arrieta et al. [22] classified studies on XAI into two approaches: some works focus on creating transparent models, while most works wrap black-box models with a layer of explainability, the so-called post-hoc models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Vale, 2022, Explainable artificial intelligence (XAI) post-hoc explainability methods: Risks and limitations in non-discrimination law, AI Ethics source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Especially in healthcare, accuracy (and to a lesser extent performance) is crucial as it could be a matter of life and death.

    “Especially in healthcare, accuracy (and to a lesser extent performance) is crucial as it could be a matter of life and death.”
    Quote in the paperNot a substantial claim
21 of 21

  • The paper relies on this without support.

    AI algorithms can be perceived as black boxes, and this limits trust in their reliability, which is a major issue in healthcare where decisions can be life-or-death.

    “However, AI has one big risk: it can be perceived as a 'black box', limiting trust in its reliability, which is a very big issue in an area in which a decision can mean life or death.”
    “They can be perceived as a 'black box': something goes in, and something comes out, with no understanding of what happens inside.”
    “This distrust in AI algorithms, their accuracy and reliability, is a very big issue in an area in which a decision could mean life or death of the patient.”
1 of 1

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Reference 31 gives conflicting conference and publication years

    Minor

    The reference identifies the proceedings as a 2020 conference but gives 2018 as the publication year.

1 of 1

Cites

Flagged first, then by citations

5 of the 78 references couldn’t be matched to a known paper.

Research works printed but not linked: 7

  1. Holzinger, A., Goebel, R., Fong, R., Moon, T., Müller, K.-R., and Samek, W. (2020, January 12–18). Explainable AI methods—A brief overview. Proceedings of the xxAI—Beyond Explainable AI: International Workshop, Held in Conjunction with ICML 2020, Vienna, Austria.
  2. Zadeh, L.A. (1979). Fuzzy Sets, Fuzzy Logic, and Fuzzy Systems: Selected Papers, World Scientific.
  3. Mencar, C., and Alonso, J.M. (2018, January 6–7). Paving the way to explainable artificial intelligence with fuzzy modeling: Tutorial. Proceedings of the Fuzzy Logic and Applications: 12th International Workshop (WILF 2018), Genoa, Italy.
  4. Fauvel, K., Masson, V., and Fromont, E. (2020). A performance-explainability framework to benchmark machine learning methods: Application to multivariate time series classifiers. arXiv.
  5. Huynh, T.D., Tsakalakis, N., Helal, A., Stalla-Bourdillon, S., and Moreau, L. (2022). Explainability-by-Design: A Methodology to Support Explanations in Decision-Making Systems. arXiv.
  6. Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv.
  7. Marcus, G. (2020). The next decade in AI: Four steps towards robust artificial intelligence. arXiv.

Cited by

Sorted by citations

Related

Authors
Hulsen, T.
Affiliations
Philips (Netherlands) (NL)
Corresponding author
Hulsen, Tim
ORCID
Hulsen, Tim
Venue
AI
Publisher
MDPI AG
Published
2023-08-10
Volume
4
Issue
3
Pages
652-666
ISSN
2673-2688 (ISSN-L)
DOI
10.3390/ai4030034
OpenAlex
W4385766743
Semantic Scholar
747ce5b1d0178cb08dcf48e43f738f1476876ed5
Cited by
305
References
78
Citations by year
2026: 92 · 2025: 112 · 2024: 85 · 2023: 16
FWCI
26.99
Type
journal article
Language
en
Open access
Licence
Locations
  • AI (publishedVersion, cc-by)
  • Preprints.org (acceptedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
explainable AI, artificial intelligence in healthcare, clinical decision support, black box problem, trust in AI, interpretability, medical image analysis, radiology report analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review