Trust in AI: why we should be designing for APPROPRIATE reliance
Open accessAbstract
Use of artificial intelligence in healthcare, such as machine learning-based predictive algorithms, holds promise for advancing outcomes, but few systems are used in routine clinical practice. Trust has been cited as an important challenge to meaningful use of artificial intelligence in clinical practice. Artificial intelligence systems often involve automating cognitively challenging tasks. Therefore, previous literature on trust in automation may hold important lessons for artificial intelligence applications in healthcare. In this perspective, we argue that informatics should take lessons from literature on trust in automation such that the goal should be to foster appropriate trust in artificial intelligence based on the purpose of the tool, its process for making recommendations, and its performance in the given context. We adapt a conceptual model to support this argument and present recommendations for future work.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Over-reliance or under-reliance on ai is wrong when decoupled from considerations on the relative informativeness of the ai.
“over-reliance or under-reliance on ai, regard a general attitude towards support systems, which is wrong when decoupled from considerations on the relative informativeness of the ai39,41,42.”
From Experimental evidence of effective human–AI collaboration in medical decision-making (2022) · cites this paper as [39]
The successful extraction of two critical task parameters (endoscopists' own accuracy and the ai's accuracy) was likely at the core of the endoscopists' ability to select and follow the best ai advice.
“The successful extraction of two critical task parameters was likely at the core of this ability: endoscopists could intuitively but reliably predict for each lesion both their accuracy (not obvious37,38) and the accuracy of the ai (not obvious39).”
From Experimental evidence of effective human–AI collaboration in medical decision-making (2022) · cites this paper as [39]
Providing information that explains how algorithms come to a prediction, thereby increasing transparency, is thought to support the critical interpretation of predictions, providing decision support and facilitating appropriate trust.
“Providing information that explains how algorithms come to a prediction, thereby increasing transparency, is thought to support the critical interpretation of predictions, providing decision support and facilitating appropriate trust.7,17”
From Appropriate trust in artificial intelligence for the optical diagnosis of colorectal polyps: the role of human/artificial intelligence interaction (2024) · cites this paper as [17]
Numerous variables influence prediction utilization, including technological characteristics such as transparency (explainable AI), usability, and interpretability, but also the stated and perceived CADx accuracy.
“Numerous variables influence prediction utilization.These include technological characteristics such as transparency (explainable AI), usability, and interpretability [13][14][15] but also the stated and perceived CADx accuracy.16,17”
From Appropriate trust in artificial intelligence for the optical diagnosis of colorectal polyps: the role of human/artificial intelligence interaction (2024) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations
GREENHALGH, T., ROBERT, G., MACFARLANE, F., BATE, P., & KYRIAKIDOU, O. (2004) · The Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00325.x · Cited by 7,448 · see citing works
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · see citing works
Assessing the Performance of Prediction Models: A Framework for Traditional and Novel Measures
Steyerberg, E. W., Vickers, A. J., Cook, N. R., Gerds, T., Gonen, M., Obuchowski, N., Pencina, M. J., & Kattan, M. W. (2010) · Epidemiology · doi:10.1097/ede.0b013e3181c30fb2 · Cited by 5,170 · see citing works
Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
Doyle, T. (2017) · The Information Society · doi:10.1080/01972243.2017.1354593 · Cited by 3,110 · see citing works
Trust in Automation: Integrating Empirical Evidence on Factors That Influence Trust
Hoff, K. A., & Bashir, M. (2015) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1177/0018720814547570 · Cited by 2,817 · see citing works
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310 · see citing works
Explainable machine-learning predictions for the prevention of hypoxaemia during surgery
Lundberg, S. M., Nair, B., Vavilala, M. S., Horibe, M., Eisses, M. J., Adams, T., Liston, D. E., Low, D. K.-W., Newman, S.-F., Kim, J., et al. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0304-0 · Cited by 1,967 · see citing works
Telerobotics, Automation, and Human Supervisory Control
Sheridan, T. B. (1992) · Medical Entomology and Zoology · Cited by 1,949 · see citing works
Trust, control strategies and allocation of function in human-machine systems
LEE, J., & MORAY, N. (1992) · Ergonomics · doi:10.1080/00140139208967392 · Cited by 1,459 · see citing works
In AI we trust? Perceptions about automated decision-making by artificial intelligence
Araujo, T., Helberger, N., Kruikemeier, S., & de Vreese, C. H. (2020) · AI & SOCIETY · doi:10.1007/s00146-019-00931-w · Cited by 1,038 · see citing works
The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations
Shneiderman, B. (2003) · The Craft of Information Visualization · doi:10.1016/b978-155860915-0/50046-9 · Cited by 515 · see citing works
SEIPS 3.0: human-centered design of the patient journey for patient safety
Carayon, P., Wooldridge, A., Hoonakker, P., Hundt, A. S., & Kelly, M. M. (2020) · Applied Ergonomics · doi:10.1016/j.apergo.2019.103033 · Cited by 493 · see citing works
Deep Learning in Medicine—Promise, Progress, and Challenges
Wang, F., Casalino, L. P., & Khullar, D. (2019) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.7117 · Cited by 483 · see citing works
Making Machine Learning Models Clinically Useful
Shah, N. H., Milstein, A., & Bagley, P. (2019) · JAMA · doi:10.1001/jama.2019.10306 · Cited by 294 · see citing works
My Friends, Editors, Algorithms, and I
Thurman, N. J., Moeller, J., Helberger, N., & Trilling, D. (2018) · Digital Journalism · doi:10.1080/21670811.2018.1493936 · Cited by 249 · see citing works
Should Health Care Demand Interpretable Artificial Intelligence or Accept “Black Box” Medicine?
Wang, F., Kaushal, R., & Khullar, D. (2020) · Annals of Internal Medicine · doi:10.7326/m19-2548 · Cited by 244 · see citing works
SEG Technical Program Expanded Abstracts 2019
(2019) · doi:10.1190/segeab.38 · Cited by 240 · see citing works
Implementing Electronic Health Care Predictive Analytics: Considerations And Challenges
Amarasingham, R., Patzer, R. E., Huesch, M. D., Nguyen, N. Q., & Xie, B. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0352 · Cited by 180 · see citing works
Oh, J., Makar, M., Fusco, C., McCaffrey, R., Rao, K. K., Ryan, E. E., Washer, L., West, L. R., Young, V. B., Guttag, J. V., et al. (2018) · Infection Control and Hospital Epidemiology · doi:10.1017/ice.2018.16 · Cited by 124 · see citing works
Improving Bayesian Reasoning: The Effects of Phrasing, Visualization, and Spatial Ability
Ottley, A., Peck, E. M., Harrison, L., Afergan, D., Ziemkiewicz, C., Taylor, H., Han, P., & Chang, R. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467758 · Cited by 115 · see citing works
Barda, A. J., Horvat, C. M., & Hochheiser, H. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01276-x · Cited by 105 · see citing works
Explainable Artificial Intelligence for Safe Intraoperative Decision Support
Gordon, L., Grantcharov, T., & Rudzicz, F. (2019) · JAMA Surgery · doi:10.1001/jamasurg.2019.2821 · Cited by 94 · see citing works
Lubo-Robles, D., Devegowda, D., Jayaram, V., Bedle, H., Marfurt, K. J., & Pranter, M. J. (2020) · Second International Meeting for Applied Geoscience & Energy · doi:10.1190/segam2020-3428275.1 · Cited by 54 · see citing works
Benda, N. C., Das, L. T., Abramson, E. L., Blackburn, K., Thoman, A., Kaushal, R., Zhang, Y., & Ancker, J. S. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa021 · Cited by 45 · see citing works
Self-citationLay individuals' perceptions of artificial intelligence ( AI )‐empowered healthcare systems
Zhang, Z., Genc, Y., Xing, A., Wang, D., Fan, X., & Citardi, D. (2020) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.326 · Cited by 37 · see citing works
Karvonen, K. (2001) · International Conference on Human-Computer Interaction · Cited by 22 · see citing works
Published models that predict hospital readmission: a critical appraisal
Liu, L. G., Rogers, J. R., Reeder, R., Walsh, C. G., Kansagara, D. L., Vawdrey, D. K., & Salmasian, H. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044964 · Cited by 17 · see citing works
Duez, P. P., Zuliani, M. J., & Jamieson, G. A. (2006) · Proceedings of CASCON · doi:10.1145/1188966.1188978 · Cited by 16 · see citing works
Incentives, expertise, and medical decisions: Testing the robustness of natural frequency framing.
Ferguson, E., & Starmer, C. (2013) · Health Psychology · doi:10.1037/a0033720 · Cited by 14 · see citing works
Reducing Imaging Utilization in Primary Care Through Implementation of a Peer Comparison Dashboard
Halpern, D. J., Clark-Randall, A., Woodall, J., et al. (2020) · Journal of General Internal Medicine · doi:10.1007/s11606-020-06164-8 · Cited by 14 · see citing works
Machine Learning and Visualization in Clinical Decision Support: Current State and Future DirectionsPreprint
Levy-Fix, G., Kuperman, G., & Elhadad, N. (2019) · arXiv.org · doi:10.48550/arxiv.1906.02664 · Cited by 12 · see citing works
Reale, C., Novak, L. L., Robinson, K., Simpson, C., Ribeiro, J. D., Franklin, J. C., Ripperger, M. A., & Walsh, C. G. (2020) · PubMed Central · Cited by 9 · see citing works
Self-citationNarrative Review and Evidence Mapping: Artificial Intelligence in Clinical Care
Girosi, F., Mann, S., & Kareddy, V. (2021) · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [12]12. Zuboff S. In the Age of the Smart Machine. New York, NY: Basic Books; 1988. [Google Scholar]
- [17]17. Deeks A. The judicial demand for explainable artificial intelligence. Columbia Law Rev 2019; 119 (7): 1829–50. [Google Scholar]
Cited by
Sorted by citations
Experimental evidence of effective human–AI collaboration in medical decision-making
Reverberi, C., Rigon, T., Solari, A., Hassan, C., Cherubini, P., GI Genius CADx Study Group, Antonelli, G., Awadie, H., Bernhofer, S., Carballal, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-18751-2 · Cited by 235 · see citing works
Ong, J. C. L., Seng, B. J. J., Law, J. Z. F., Low, L. L., Kwa, A. L. H., Giacomini, K. M., & Ting, D. S. W. (2024) · Cell Reports Medicine · doi:10.1016/j.xcrm.2023.101356 · Cited by 111 · see citing works
Salwei, M. E., & Carayon, P. (2022) · Journal of Cognitive Engineering and Decision Making · doi:10.1177/15553434221097357 · Cited by 112 · see citing works
Psychological Traits and Appropriate Reliance: Factors Shaping Trust in AI
Küper, A., & Krämer, N. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2348216 · Cited by 49 · see citing works
Leveraging large language models to foster equity in healthcare
Rodriguez, J. A., Alsentzer, E., & Bates, D. W. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae055 · Cited by 32 · see citing works
van der Zander, Q. E. W., Roumans, R., Kusters, C. H. J., Dehghani, N., Masclee, A. A. M., de With, P. H. N., van der Sommen, F., Snijders, C. C. P., & Schoon, E. J. (2024) · Gastrointestinal Endoscopy · doi:10.1016/j.gie.2024.06.029 · Cited by 27 · see citing works
Küper, A., Lodde, G. C., Livingstone, E., Schadendorf, D., & Krämer, N. (2025) · Journal of Medical Internet Research · doi:10.2196/58660 · Cited by 31 · see citing works
“Research ranges from system acceptance and adoption [39,40] to specific investigations of trust in systems [41-44] to the calibration of trust and the appropriateness of reliance [8,9,45-49].”
- Authors
- Benda, N. C., Novak, L. L., Reale, C., & Ancker, J. S.
- Affiliations
- Weill Cornell Medicine (US)
- Vanderbilt University Medical Center (US)
- Corresponding author
- Benda, Natalie C
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-11-02
- Volume
- 29
- Issue
- 1
- Pages
- 207-212
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC8714273
- OpenAlex
- W3210146206
- Cited by
- 76
- References
- 33
- Citations by year
- 2026: 25 · 2025: 15 · 2024: 19 · 2023: 11 · 2022: 6
- FWCI
- 2.62
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
- Keywords
- trust in AI, trust in automation, clinical decision support, predictive algorithms, clinical implementation of AI, design for trust, conceptual model, human-AI interaction
- MeSH terms
- Machine Learning, Algorithms, Artificial Intelligence, Automation, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
