Physician understanding, explainability, and trust in a hypothetical machine learning risk calculator
Open accessAbstract
Objective Implementation of machine learning (ML) may be limited by patients' right to "meaningful information about the logic involved" when ML influences healthcare decisions. Given the complexity of healthcare decisions, it is likely that ML outputs will need to be understood and trusted by physicians, and then explained to patients. We therefore investigated the association between physician understanding of ML outputs, their ability to explain these to patients, and their willingness to trust the ML outputs, using various ML explainability methods. Materials and methods We designed a survey for physicians with a diagnostic dilemma that could be resolved by an ML risk calculator. Physicians were asked to rate their understanding, explainability, and trust in response to 3 different ML outputs. One ML output had no explanation of its logic (the control) and 2 ML outputs used different model-agnostic explainability methods. The relationships among understanding, explainability, and trust were assessed using Cochran-Mantel-Haenszel tests of association. Results The survey was sent to 1315 physicians, and 170 (13%) provided completed surveys. There were significant associations between physician understanding and explainability (P Conclusions Physician understanding, explainability, and trust in ML risk calculators are related. Physicians preferred ML outputs accompanied by model-agnostic explanations but the explainability method did not alter intended physician behavior.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Physicians and nurses rate explainability in terms of patient data and personal understanding as highly related to trust in AI.
“Physicians and nurses rate explainability in terms of patient data and personal understanding as highly related to trust in AI;73,77 however, current methods of AI explainability have been found to have limited usefulness in practice.78”
From Potential uses of AI for perioperative nursing handoffs: a qualitative study (2023) · cites this paper as [77]
When the logic behind predictions was not understood, physicians did not trust them.
“When the logic behind the predictions was not understood, the physicians did not trust them.”
From Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study (2022) · cites this paper as [6]
Understandability and explainability are positively associated with physicians' trust in a machine learning-based CDSS.
“In a study of physicians’ ability to understand and explain a machine learning–based CDSS’s logic, Diprose et al [6] found that both understandability and explainability were positively associated with trust.”
From Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study (2022) · cites this paper as [6]
Most research on clinician trust has focused on physicians' trust in predictive CDSSs and machine learning-based diagnostic CDSSs.
“Moreover, most research on clinician trust has focused on physicians’ trust in predictive CDSSs and machine learning–based diagnostic CDSSs [2,6,15].”
From Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study (2022) · cites this paper as [6]
Prior studies on explainability and trust in predictive CDSSs were conducted using simulated rather than live implemented predictive CDSSs.
“The latter two were conducted by referring to simulated as opposed to live implemented predictive CDSSs.”
From Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study (2022) · cites this paper as [6]
Cites
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
The EU General Data Protection Regulation (GDPR)
Voigt, P., & von dem Bussche, A. (2017) · doi:10.1007/978-3-319-57959-7 · Cited by 3,001 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Petersson, L., Larsson, I., Nygren, J. M., Nilsen, P., Neher, M., Reed, J. E., Tyskbo, D., & Svedberg, P. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08215-8 · Cited by 434 · see citing works
Čartolovni, A., Tomičić, A., & Lazić Mosler, E. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104738 · Cited by 267 · see citing works
Gama, F., Tyskbo, D., Nygren, J., Barlow, J., Reed, J., & Svedberg, P. (2022) · Journal of Medical Internet Research · doi:10.2196/32215 · Cited by 216 · see citing works
Explainable AI for clinical and remote health applications: A survey on tabular and time series data
Di Martino, F., & Delmastro, F. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-022-10304-3 · Cited by 216 · see citing works
Bell, A., Solano-Kamaiko, I., Nov, O., & Stoyanovich, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533090 · Cited by 103 · see citing works
Schwartz, J. M., George, M., Rossetti, S. C., Dykes, P. C., Minshall, S. R., Lucas, E., & Cato, K. D. (2022) · JMIR Human Factors · doi:10.2196/33960 · Cited by 88 · see citing works
“This preference may differ for machine learning–based diagnostic CDSSs, as hypothesized by Diprose et al [6].”
BackgroundInfluentialGuiding principles for the responsible development of artificial intelligence tools for healthcare
Badal, K., Lee, C. M., & Esserman, L. J. (2023) · Communications Medicine · doi:10.1038/s43856-023-00279-9 · Cited by 77 · see citing works
Explainable machine learning practices: opening another black box for reliable medical AI
Ratti, E., & Graves, M. (2022) · AI and Ethics · doi:10.1007/s43681-022-00141-z · Cited by 61 · see citing works
Stidham, R. W., Cai, L., Cheng, S., Rajaei, F., Hiatt, T., Wittrup, E., Rice, M. D., Bishu, S., Wehkamp, J., Schultz, W., et al. (2024) · Gastroenterology · doi:10.1053/j.gastro.2023.09.049 · Cited by 63 · see citing works
Laukka, E., Hammarén, M., & Kanste, O. (2022) · Journal of Nursing Management · doi:10.1111/jonm.13769 · Cited by 44 · see citing works
Brankovic, A., Cook, D., Rahman, J., Delaforce, A., Li, J., Magrabi, F., Cabitza, F., Coiera, E., & Bradford, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01764-2 · Cited by 37 · see citing works
Potential uses of AI for perioperative nursing handoffs: a qualitative study
King, C. R., Shambe, A., & Abraham, J. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad015 · Cited by 27 · see citing works
“Physicians and nurses rate explainability in terms of patient data and personal understanding as highly related to trust in AI; 73,77 however, current methods of AI explainability have been found to have limited usefulness in practice.”
BackgroundUser-centered design of a machine learning dashboard for prediction of postoperative complications
Fritz, B. A., Pugazenthi, S., Budelier, T. P., Tellor Pennington, B. R., King, C. R., Avidan, M. S., & Abraham, J. (2024) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000006577 · Cited by 15 · see citing works
Explainable artificial intelligence for multi-modal cancer analysis: from genomics to immunology
Zhang, M., & Yin, L. (2026) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2025.105040 · Cited by 5 · see citing works
- Authors
- Diprose, W. K., Buist, N., Hua, N., Thurier, Q., Shand, G., & Robinson, R.
- Affiliations
- University of Auckland (NZ)
- Northland District Health Board (NZ)
- Orion Health, Auckland, New Zealand
- Waitemata District Health Board (NZ)
- Corresponding author
- Diprose, William K
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-02-27
- Volume
- 27
- Issue
- 4
- Pages
- 592-600
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7647292
- OpenAlex
- W3007542085
- Cited by
- 188
- References
- 2
- Citations by year
- 2026: 19 · 2025: 34 · 2024: 40 · 2023: 38 · 2022: 34 · 2021: 17 · 2020: 6
- FWCI
- 6.50
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Precision Driven Health
- Auckland Regional Health Boards
- University of Auckland
- Orion Health
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- machine learning, explainability, trust in physicians, risk calculator, model-agnostic explanation, patient communication, clinical decision support, algorithmic transparency
- MeSH terms
- Machine Learning, Artificial Intelligence, Attitude of Health Personnel, Attitude to Computers, Decision Making, Computer-Assisted, Humans, Physicians, Pulmonary Embolism, Surveys and Questionnaires, User-Computer Interface, Risk Assessment, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
