The Role of XAI in Advice-Taking from a Clinical Decision Support System: A Comparative User Study of Feature Contribution-Based and Example-Based Explanations
Open accessThe DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index.
Abstract
Explainable artificial intelligence (XAI) has shown benefits in clinical decision support systems (CDSSs); however, it is still unclear to CDSS developers how to select an XAI method to optimize the advice-taking of healthcare practitioners. We performed a user study on healthcare practitioners based on a machine learning-based CDSS for the prediction of gestational diabetes mellitus to explore and compare two XAI methods: explanation by feature contribution and explanation by example. Participants were asked to make estimates for both correctly and incorrectly predicted cases to determine if there were any over-reliance or self-reliance issues. We examined the weight of advice and healthcare practitioners’ preferences. Our results based on statistical tests showed no significant difference between the two XAI methods regarding the advice-taking. The CDSS explained by either method had a substantial impact on the decision-making of healthcare practitioners; however, both methods may lead to over-reliance issues. We identified the inclination towards CDSS use as a key factor in the advice-taking from an explainable CDSS among obstetricians. Additionally, we found that different types of healthcare practitioners had differing preferences for explanations; therefore, we suggest that CDSS developers should select XAI methods according to their target users.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This oversimplification and an high enough trust on the users side can lead to over reliance [11], in which the users trust the model to much and let it cloud their judgement.”
From Study on the Helpfulness of Explainable Artificial Intelligence (2024) · cites this paper as [11]
Citations
Cites
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
Gauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Humans and automation: Use, misuse, disuse, abuse
Parasuraman, R., & Riley, V. (1997) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/001872097778543886 · Cited by 4,554
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
An overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Receiving other people’s advice: Influence and benefit
Yaniv, I. (2004) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2003.08.002 · Cited by 652
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629
Self-citationTaking Advice: Accepting Help, Improving Judgment, and Sharing Responsibility
Harvey, N., & Fischer, I. (1997) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1997.2697 · Cited by 588
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441
Cited by
Gomez, C., Cho, S. M., Ke, S., Huang, C.-M., & Unberath, M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1521066 · Cited by 209
“In each pattern formulation, we considered an appropriate level of abstraction so that they can capture multiple actions of the agents and generalize over various studies included in the systematic review.”
Explainable Artificial Intelligence-Based Decision Support Systems: A Recent Review
Kostopoulos, G., Davrazos, G., & Kotsiantis, S. B. (2024) · Electronics · doi:10.3390/electronics13142842 · Cited by 267
“[16] x x [40,81,82] x [50] x x [51] x x x x x [52,54,57,58] x [62,69,70,76] x [53] x [55,56,59] x [60,78] x [61,68] x x [63] x x [64] x x [65] x x [20,66,72,73] x [75] x x x x x [77] x [79,80] x [83] x x”
Dey, M. T. (2024) · Advances in environmental engineering and green technologies book series · doi:10.4018/979-8-3693-7822-9.ch012 · Cited by 137
Hur, S., Lee, Y., Park, J., Jeon, Y. J., Cho, J. H., Cho, D., Lim, D., Hwang, W., Cha, W. C., & Yoo, J. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01958-8 · Cited by 118
“Moreover, there is no consensus on how explanation formats affect clinicians ’ acceptance to AI suggestions 13,20 .”
The Impact of Imperfect XAI on Human-AI Decision-Making
Morrison, K., Spitzer, P., Turri, V., Feng, M., Kühl, N., & Perer, A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3641022 · Cited by 60
“00, example-based: p-value = 0 .”
BackgroundAn, Y., Du, H., Ma, S., Niu, Y., Liu, D., Wang, J., Du, Y., Childs, C. J., Walsh, J. J., & Dong, R. (2023) · Earth-Science Reviews · doi:10.1016/j.earscirev.2023.104509 · Cited by 71
Self-citationAgriculture 5.0 and Explainable AI for Smart Agriculture: A Scoping Review
Razak, A., Yogarayan, S., Sayeed, M. S., & Derafi, M. I. F. M. (2024) · Emerging Science Journal · doi:10.28991/esj-2024-08-02-024 · Cited by 91
“The synergy between XAI techniques and real-time data from IoT devices and sensors, facilitated by Agriculture 5.0, ensures that the decisions made by AI models are understandable and justifiable by the users [94].”
MethodInfluentialXu, X., Wu, F., Bilal, M., Xia, X., Dou, W., Yao, L., & Zhong, W. (2024) · Science China Information Sciences · doi:10.1007/s11432-023-3987-y · Cited by 70
Explainable AI in agriculture: review of applications, methodologies, and future directions
Pai, D. G., Balachandra, M., & Kamath, R. (2025) · Engineering Research Express · doi:10.1088/2631-8695/ae086a · Cited by 93
Machine learning-based clinical decision support systems for pregnancy care: A systematic review
Du, Y., McNestry, C., Wei, L., Antoniadi, A. M., McAuliffe, F. M., & Mooney, C. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105040 · Cited by 70
Self-citationKim, M., Kim, S., Kim, J., Song, T.-J., & Kim, Y. (2024) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2023.103160 · Cited by 53
Lukkien, D., Stolwijk, N. E., Askari, S. I., Hofstede, B. M., Nap, H. H., Boon, W., Peine, A., Moors, E. H. M., & Minkman, M. M. N. (2024) · JMIR Nursing · doi:10.2196/55962 · Cited by 69
“…to be used and data to be collected [74], the processing of these data by AI, who gets access to the data and AI-based insights, the explanation of AI-based insights to users [75], and the extent to which AI-DSSs proactively advise caregivers about care needs and strategies (see prerequisite 4).”
MethodNicolson, A., Bradburn, E., Gal, Y., Papageorghiou, A. T., & Noble, J. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02023-0 · Cited by 57
Explainable AI decision support improves accuracy during telehealth strep throat screening
Gomez, C., Smith, B.-L., Zayas, A., Unberath, M., & Canares, T. (2024) · Communications Medicine · doi:10.1038/s43856-024-00568-x · Cited by 30
Bergomi, L., Nicora, G., Orlowska, M. A., Podrecca, C., Bellazzi, R., Fregosi, C., Salinaro, F., Bonzano, M., Crescenzi, G., Speciale, F., et al. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03045-0 · Cited by 33
“Like other previous similar studies [16, 18, 20, 38], we acknowledge the limited sample size (10 clinicians) as one of the main limitations of this work, and we recognize that the findings cannot be broadly generalized across all specialties or clinical settings.”
Aliyeva, K. R., & Mehdiyev, N. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119987 · Cited by 31
Eckhardt, S., Kühl, N., Dolata, M., & Schwabe, G. (2025) · ACM Computing Surveys · doi:10.1145/3776528 · Cited by 44
Computational complexity in explainable decision support system: A review
Ezeji, I. N., Adigun, M. O., & Oki, O. A. (2024) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-219407 · Cited by 22
Roychowdhury, S., Lanfranchi, V., & Mazumdar, S. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110944 · Cited by 21
Towards an explainable clinical decision support system for large-for-gestational-age births
Du, Y., Rafferty, A. R., McAuliffe, F. M., Mehegan, J., & Mooney, C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281821 · Cited by 11
Self-citationStudy on the Helpfulness of Explainable Artificial Intelligence
Labarta, T., Kulicheva, E., Froelian, R., Geißler, C., Melman, X., & von Klitzing, J. (2024) · Communications in Computer and Information Science · doi:10.1007/978-3-031-63803-9_16 · Cited by 8
“This oversimplification and an high enough trust on the users side can lead to over reliance [11], in which the users trust the model to much and let it cloud their judgement.”
BackgroundNap, H. H., Stolwijk, N. E., Askari, S. I., Lukkien, D., Hofstede, B. M., Morresi, N., Casaccia, S., Amabili, G., Bevilacqua, R., Margaritini, A., et al. (2024) · Frontiers in Dementia · doi:10.3389/frdem.2024.1400624 · Cited by 13
Explainable AI in Disease Diagnosis
Bedi, P., Thukral, A., & Dhiman, S. (2024) · Computational intelligence methods and applications · doi:10.1007/978-981-97-3705-5_5 · Cited by 8
Imperfections of XAI: Phenomena Influencing AI-Assisted Decision-Making
Spitzer, P., Morrison, K., Turri, V., Feng, M., Perer, A., & Kühl, N. (2025) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3750052 · Cited by 17
RIXA - Explaining Artificial Intelligence in Natural Language
Becker, M., Vishwesh, V., Birnstill, P., Schwall, F., Wu, S., & Beyerer, J. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00118 · Cited by 5
“However, the challenge arises when one notes that most of the XAI implementations lean too heavily on the technical side, often overwhelming end users with intricate metrics and confusing data visualizations [12]– [14].”
BackgroundJung, I.-C., Zerlik, M., Schuler, K., Sedlmayr, M., & Sedlmayr, B. (2026) · JMIR Formative Research · doi:10.2196/77481 · Cited by 5
“The positive assessment of the feature importance explanation concept for the physician group is in line with other research efforts [14,15].”
An Explainable Machine Learning Approach for Early Prediction of Gestational Diabetes Mellitus (GDM)
Aatif, M., Islam, I. U., Iltaf, N., & Afzal, H. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-89813-6_11 · Cited by 2
XAINT: eXplainable AI through Iterative Network Truncation
Coppari, A., Capra, L., & Carlevaro, C. (2025) · IEEE Symposium on Trustworthy, Explainable and Responsible Computational Intelligence (CITREx) · doi:10.1109/citrex64975.2025.10974946 · Cited by 1
“Several XAI applications have been developed for sensitive domains like healthcare [8], [9], [10] and cybersecurity [11], [12].”
Evaluation of Post-Hoc Explainability Methods for Glaucoma Classification Using Fundus Images
Ming, H. K., Vong, W.-T., Loh, B. C. S., & Then, P. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-92605-1_40 · Cited by 1
Domain Experience and Expertise in Explainable AI Applications: A Bearing Fault Diagnosis Case Study
Zhao, Z., Castelle, M., & Turkay, Ç. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757583 · Cited by 3
“While healthcare-focused XAI studies note practitioners’ preferences [34, 43], links between experience, expertise, and AI explanation adoption remain unclear.”
It’s Not Just for Trust: Designing for Emerging Uses of Explainable AI in Clinical Decision-Making
Morrison, K., Li, Z., Kim, M. J., Hong, C. (., Kraisangka, J., Correa‐Jaque, P., Fauvel, C., Sahay, S., Vanderpool, R. R., Everett, A., et al. (2026) · ACM Transactions on Computing for Healthcare · doi:10.1145/3815175 · Cited by 1
Yang, M. H., Jiang, J., Liu, X., & Kahai, S. S. (2026) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2026.103898 · Cited by 1
Ali, H., Farooq, M., & Shah, U. (2025) · Journal of Artificial Intelligence for Healthcare Systems · doi:10.68159/g968708531 · Cited by 1
Lialiou, P., & Maglogiannis, I. (2026) · Research Square · doi:10.21203/rs.3.rs-9055427/v1 · Cited by 1
Lukkien, D., Stolwijk, N. E., Askari, S. I., Hofstede, B. M., Nap, H. H., Boon, W., Peine, A., Moors, E. H. M., & Minkman, M. M. N. (2024) · doi:10.2196/preprints.55962
Review of Applied Science and TechnologyPreprint
RAHMAN, M. A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5360313 · Cited by 1
Review of Applied Science and TechnologyPreprint
Nuruzzaman, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5371694 · Cited by 3
Designing and Personalising Hybrid Health Explanations for Lay Users
Szymanski, M., Keyaerts, S., Conati, C., Croon, R. D., Abeele, V. V., & Verbert, K. (2025) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3772071 · Cited by 1
“Feature importances can be used to explain clinical decision making and may be visually represented through SHAP or LIME (e.g., assessing risk for gestational diabetes mellitus [24], assessing COVID severity [95], or predicting the quality of life of ALS patients [4]).”
Jung, J.-Y., Yang, W., & Kim, J.-J. (2026) · doi:10.2196/preprints.92831
Zhan, M. (2025) · International Conference on Smart Healthcare and Wearable Intelligent Devices (SHWID) · doi:10.1145/3788112.3788141
Zahid, M. M. R., Tanzim, M. B. M., Rahman, M. S., Biswas, S., Jinia, J. T., & Bijoy, M. H. I. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-15346-3_16
The HEART Interface: Visualizing Risk Score Uncertainty in the Cardiothoracic ICU
Nourani, M., Nguyen, L., Barry, C. L., Jin, Q., & Tory, M. K. (2026) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3742413.3789109 · Cited by 2
An XAI Framework for Heart Disease Oriented Healthcare Decision Support Systems
Kamdi, N. G., Dagadkar, S. N., Puri, C., Bansod, S., Kamdi, P., & Pimpalkar, A. K. (2026) · IEEE International Conference on Intelligent Technologies for a Sustainable and Inclusive Future (ICITSIF) · doi:10.1109/icitsif69060.2026.11608423
“Du et al. [20] compared feature-based and example-based explanations to examine the role of XAI in CDSS advice-taking.”
Jung, J.-Y., Yang, W., & Kim, J.-J. (2026) · JMIR Medical Informatics · doi:10.2196/92831
Frey, N., Agha-Mir-Salim, L., Renner, L., Haufe, S., Voge, L., & Balzer, F. (2026) · Research Square · doi:10.21203/rs.3.rs-10892043/v1 · Cited by 2
Lee, J., Bae, J., Ryu, E. S., Kim, L., Kim, D., Lee, S., & Yun, M. H. (2026) · Ergonomics · doi:10.1080/00140139.2026.2731219
Evaluating Explainable AI with Humans: Trust, Reliance, Understanding, and Performance
Frez, J., Baloian, N. A., Ahumada, L., & Harutyunyan, A. N. (2026) · ACM Computing Surveys · doi:10.1145/3856819
“…[127], [121], [64], [71], [20], [41], [86], [80], [85], [114], [2], [17], [96], [125], [99] Worklow, validation, and accountability 2022ś2025; = 6 [36], [43], [16], [9], [57], [34] Placebo, salience, and accuracy confounds 2019ś2025; = 20 [129], [122], [26], [21], [132], [110], [115], [95], [46],…”
Metadata
- Authors
- Du, Y., Antoniadi, A. M., McNestry, C., McAuliffe, F. M., & Mooney, C.
- Affiliations
- University College Dublin (IE)
- Corresponding author
- Mooney, Catherine
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2022-10-13
- Volume
- 12
- Issue
- 20
- Pages
- 10323
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4306149475
- Cited by
- 47
- References
- 35
- Citations by year
- 2026: 12 · 2025: 17 · 2024: 12 · 2023: 6
- FWCI
- 2.88
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- University College Dublin and National Maternity Hospital Medical Fund
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- explainable AI, clinical decision support systems, advice taking, example-based explanations, overreliance on AI, obstetricians
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.