The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Assessing the communication gap between AI models and healthcare professionals: Explainability, utility and trust in AI-driven clinical decision-making
Open accessAbstract
This paper contributes with a pragmatic evaluation framework for explainable Machine Learning (ML) models for clinical decision support. The study revealed a more nuanced role for ML explanation models, when these are pragmatically embedded in the clinical context. Despite the general positive attitude of healthcare professionals (HCPs) towards explanations as a safety and trust mechanism, for a significant set of participants there were negative effects associated with confirmation bias, accentuating model over-reliance and increased effort to interact with the model. Also, contradicting one of its main intended functions, standard explanatory models showed limited ability to support a critical understanding of the limitations of the model. However, we found new significant positive effects which repositions the role of explanations within a clinical context: these include reduction of automation bias, addressing ambiguous clinical cases (cases where HCPs were not certain about their decision) and support of less experienced HCPs in the acquisition of new domain knowledge.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The intuitiveness of model explanations varies substantially with respect to end users.
“the intuitiveness of model explanations varies substantially with respect to end users [35-37]”
From Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review (2026) · cites this paper as [35]
Interpreting explanations primarily through the lens of existing medical literature may therefore introduce confirmation bias, whereby explanations that align with prior expectations are accepted uncritically while alternative patterns are overlooked.
“Interpreting explanations primarily through the lens of existing medical literature may therefore introduce confirmation bias, whereby explanations that align with prior expectations are accepted uncritically while alternative patterns are overlooked [35].”
From Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review (2026) · cites this paper as [35]
Explanations that focus on algorithmic mechanisms poorly convey information in terms of established diagnostic criteria or clinical reasoning processes.
“Explanations that focus on algorithmic mechanisms poorly convey information in terms of established diagnostic criteria or clinical reasoning processes [35].”
From Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review (2026) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Gauriat, C.-M., Pencolé, Y., Ribot, P., et al. (2024) · HAL · doi:10.4230/oasics.dx.2024.27 · Cited by 12,623
An overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Definitions, methods, and applications in interpretable machine learning
Murdoch, W. J., Singh, C., Kumbier, K., Abbasi-Asl, R., & Yu, B. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1900654116 · Cited by 2,167 · cited 3×
“Explainability [1, 2, 3, 4, 5, 6] emerged as a research area aiming to address interpretability bottlenecks in ML models resulting from the fact that many of them can be effective in predicting an outcome, but not in explaining their underlying reasoning, which limits their practical application in critical areas.”
InfluentialAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861
The false hope of current approaches to explainable artificial intelligence in health care
Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00208-9 · Cited by 1,724
Explainable AI: Interpreting, Explaining and Visualizing Deep Learning
(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6 · Cited by 1,377
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Artificial Intelligence and Human Trust in Healthcare: Focus on Clinicians
Asan, O., Bayrak, A. E., & Choudhury, A. (2020) · Journal of Medical Internet Research · doi:10.2196/15154 · Cited by 825
Artificial Intelligence and Black‐Box Medical Decisions: Accuracy versus Explainability
London, A. J. (2019) · Hastings Center Report · doi:10.1002/hast.973 · Cited by 801
Does automation bias decision-making?
SKITKA, L. J., MOSIER, K. L., & BURDICK, M. (1999) · International Journal of Human-Computer Studies · doi:10.1006/ijhc.1999.0252 · Cited by 694 · cited 2×
“confirmation bias [32] and automation bias [33, 34].”
Automation bias and verification complexity: a systematic review
Lyell, D., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw105 · Cited by 451
Evaluating XAI: A comparison of rule-based and example-based explanations
van der Waa, J., Nieuwburg, E., Cremers, A., & Neerincx, M. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2020.103404 · Cited by 393
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 1×
“Although explainable ML is advocated as the methodological silver bullet for addressing transparency issues [1], some user studies, similarly to this paper, show evidence of a limited or even no impact of the explanation on performance in a task [54, 18, 55, 56, 57].”
Effects of Imperfect Automation on Decision Making in a Simulated Command and Control Task
Rovira, E. M., McGarry, K. A., & Parasuraman, R. (2007) · Human Factors The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872007779598082 · Cited by 289
Faithful and Customizable Explanations of Black Box Models
Lakkaraju, H., Kamar, E. S., Caruana, R., & Leskovec, J. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314229 · Cited by 268
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Lakkaraju, H., & Bastani, O. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375833 · Cited by 241
Problems with Shapley-value-based explanations as feature importance measures
Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020) · arXiv · doi:10.48550/arxiv.2002.11097 · Cited by 228 · cited 1×
“One of the downsides of feature importance is that it may lead to confirmation bias, both in the model derivation phase [37] and in the post-hoc interpretation of the output.”
Holzinger, A., Dehmer, M., Emmert‐Streib, F., Cucchiara, R., Augenstein, I., Ser, J. D., Samek, W., Jurišica, I., & Díaz-Rodríguez, N. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.10.007 · Cited by 223
Explainable and Interpretable Models in Computer Vision and Machine Learning
Escalante, H. J., Escalera, S., Guyon, I., Baró, X., Güçlütürk, Y., Güçlü, U., & van Gerven, M. (2018) · The Springer series on challenges in machine learning · doi:10.1007/978-3-319-98131-4 · Cited by 213
Kenny, E. M., Ford, C., Quinn, M., & Keane, M. T. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103459 · Cited by 211
Clinical Decision-Support Systems
Musen, M. A., Middleton, B., & Greenes, R. A. (2021) · Springer eBooks · doi:10.1007/978-3-030-58721-5_24 · Cited by 166
Automation bias: Empirical results assessing influencing factors
Goddard, K., Roudsari, A., & Wyatt, J. C. (2014) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2014.01.001 · Cited by 163
The Impact of Placebic Explanations on Trust in Intelligent Systems
Eiband, M., Buschek, D., Kremer, A., & Hussmann, H. (2019) · Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290607.3312787 · Cited by 152
The explainability paradox: Challenges for xAI in digital pathology
Evans, T., Retzlaff, C. O., Geißler, C., Kargl, M., Plass, M., Müller, H., Kiehl, T.-R., Zerbe, N., & Holzinger, A. (2022) · Future Generation Computer Systems · doi:10.1016/j.future.2022.03.009 · Cited by 135
I can do better than your AI: expertise and explanations
Schaffer, J., O'Donovan, J., Michaelis, J., Raglin, A., & Höllerer, T. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302308 · Cited by 119
Toward Human–AI Interfaces to Support Explainability and Causability in Medical AI
Holzinger, A., & Müller, H. (2021) · Computer · doi:10.1109/mc.2021.3092610 · Cited by 99
Barriers and Facilitators to Clinical Decision Support Systems Adoption: A Systematic Review
Devaraj, S., SHARMA, S. K., Fausto, D. J., Viernes, S., & Kharrazi, H. H. K. (2014) · Journal of Business Administration Research · doi:10.5430/jbar.v3n2p36 · Cited by 98
Nurses’ use of computerised clinical decision support systems: a case site analysis
Dowding, D., Mitchell, N., Randell, R., Foster, R., Lattimer, V., & Thompson, C. A. (2009) · Journal of Clinical Nursing · doi:10.1111/j.1365-2702.2008.02607.x · Cited by 92
Müller, H., Holzinger, A., Plass, M., Brčić, L., Stumptner, C., & Zatloukal, K. (2022) · New Biotechnology · doi:10.1016/j.nbt.2022.05.002 · Cited by 87
Evaluation of artificial intelligence on a reference standard based on subjective interpretation
Chen, P.-H. C., Mermel, C. H., & Liu, Y. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00216-8 · Cited by 63
Feature-Based Explanations Don't Help People Detect Misclassifications of Online Toxicity
Carton, S., Mei, Q., & Resnick, P. (2020) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v14i1.7282 · Cited by 58
Automation bias - a hidden issue for clinical decision support system use.
Goddard, K. S., Roudsari, A., & Wyatt, J. C. (2011) · PubMed · Cited by 43
Human-Computer Interaction - INTERACT 2017
Bernhaupt, R., Dalvi, G., Joshi, A., Balkrishan, D. K., O’Neill, J., & Winckler, M. A. A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67744-6 · Cited by 38
Ha, T., Sah, Y. J., Park, Y., & Lee, S. (2020) · Behaviour and Information Technology · doi:10.1080/0144929x.2020.1846789 · Cited by 33
Lee, R. J., Wysocki, O., Bhogal, T., Shotton, R., Tivey, A., Angelakas, A., Aung, T., Banfill, K., Baxter, M. A., Boyce, H., et al. (2021) · ESMO Open · doi:10.1016/j.esmoop.2020.100005 · Cited by 18
Self-citationRamani, D., Soh, M. Y., Merkebu, J., et al. (2020) · Diagnosis · doi:10.1515/dx-2020-0019 · Cited by 16
Burke, H., Freeman, A. T., O’Regan, P., Wysocki, O., Freitas, A., Dushianthan, A., Celinski, M., Batchelor, J., Phan, H. T. T., Borca, F., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-050331 · Cited by 16
Self-citationFreeman, A., Watson, A., O'Regan, P., Wysocki, O., Burke, H., Freitas, A., Livingstone, R., Dushianthan, A., Celinski, M., Batchelor, J., et al. (2021) · Journal of Clinical Virology · doi:10.1016/j.jcv.2021.105031 · Cited by 14
Self-citationExpl(Ai)Ned: The Impact of Explainable Artificial Intelligence on Cognitive ProcessesPreprint
Bauer, K., von Zahn, M., & Hinz, O. F. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3872711 · Cited by 13
Lee, R. J., Wysocki, O., Zhou, C., Shotton, R., Tivey, A., Lever, L., Woodcock, J., Albiges, L. K., Angelakas, A., Arnold, D., et al. (2022) · JCO Clinical Cancer Informatics · doi:10.1200/cci.21.00177 · Cited by 13
Self-citationInformation Systems and Neuroscience
Davis, F. D., Riedl, R., Brocke, J. V., et al. (2022) · Lecture notes in information systems and organisation · doi:10.1007/978-3-031-13064-9 · Cited by 7
Cognitive Properties of Representations: A Framework
Cheng, P., García, G. G., Raggi, D., Stockdill, A., & Jamnik, M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86062-2_43 · Cited by 6
Raggi, D., Stockdill, A., Jamnik, M., et al. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-54249-8_11 · Cited by 6
Wysocki, O., Zhou, C., Rogado, J., et al. (2022) · Cancers · doi:10.3390/cancers14163931 · Cited by 2
Self-citationLee, R., Wysocki, O., Zhou, C., et al. (2021) · Journal of Clinical Oncology · doi:10.1200/jco.2021.39.15_suppl.1505 · Cited by 1
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- Samek, 2019, Towards explainable Artificial Intelligence, vol. 11700, 5
- Hudon, 2021, Explainable Artificial Intelligence (XAI): how the visualization of AI predictions affects user cognitive load and confidence, vol. 52, 237
- Zhou, 2017, Effects of uncertainty and cognitive load on user trust in predictive decision making, vol. 10516, 23
Cited by
Sorted by citations
Bias recognition and mitigation strategies in artificial intelligence healthcare applications
Hasanzadeh, F., Josephson, C. B., Waters, G., Adedinsewo, D., Azizi, Z., & White, J. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01503-7 · Cited by 320
Human-centered evaluation of explainable AI applications: a systematic review
Kim, J., Maathuis, H., & Sent, D. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1456486 · Cited by 130
Probabilistic medical predictions of large language models
Gu, B., Desai, R. J., Lin, K. J., & Yang, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01366-4 · Cited by 50
Exploring explainable AI features in the vocal biomarkers of lung disease
Chen, Z., Liang, N., Li, H., Zhang, H., Li, H., Yan, L., Hu, Z., Chen, Y., Zhang, Y., Wang, Y., et al. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108844 · Cited by 26
How Do We Assess the Trustworthiness of AI? Introducing the Trustworthiness Assessment Model (TrAM)
Schlicker, N., Baum, K., Uhde, A., Sterz, S., Hirsch, M. C., & Langer, M. (2022) · doi:10.31234/osf.io/qhwvx · Cited by 79
Rosbach, E., Ammeling, J., Krügel, S., Kießig, A., Fritz, A., Ganz, J., Puget, C., Donovan, T., Klang, A., Köller, M. C., et al. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713319 · Cited by 18
Raees, M., Khan, V.-J., Lykourentzou, I., & Papangelis, K. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791467 · Cited by 5
Explainable and Interpretable AI for Voice and Speech Analysis in Clinical Care: Systematic Review
Ebraheem, M., Toghranegar, J., Bridge2AI-Voice Consortium, Bensoussan, Y., & Templeton, J. M. (2026) · Journal of Medical Internet Research · doi:10.2196/83790
Metadata
- Authors
- Wysocki, O., Davies, J. K., Vigo, M., Armstrong, A. C., Landers, D., Lee, R., & Freitas, A.
- Affiliations
- University of Manchester (GB)
- Corresponding author
- Wysocki, Oskar
- Venue
- Artificial Intelligence
- Publisher
- Elsevier BV
- Published
- 2023-03
- Volume
- 316
- Pages
- 103839
- ISSN
- 0004-3702 (ISSN-L)
- PMC
- PMC7618637
- OpenAlex
- W4312085331
- Cited by
- 161
- References
- 51
- Citations by year
- 2026: 43 · 2025: 53 · 2024: 45 · 2023: 20
- FWCI
- 9.79
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Artificial Intelligence (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Research Explorer (The University of Manchester) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Christie Charity
- European Union's Research and Innovation (965397)
- National Institute for Health and Care Research
- Horizon 2020
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- explainable machine learning, clinical decision support, healthcare workers, trust in AI, model explainability, confirmation bias, automation bias, domain knowledge acquisition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
