Trust and medical AI: the challenges we face and the expertise needed to overcome them
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) is increasingly of tremendous interest in the medical field. How-ever, failures of medical AI could have serious consequences for both clinical outcomes and the patient experience. These consequences could erode public trust in AI, which could in turn undermine trust in our healthcare institutions. This article makes 2 contributions. First, it describes the major conceptual, technical, and humanistic challenges in medical AI. Second, it proposes a solution that hinges on the education and accreditation of new expert groups who specialize in the development, verification, and operation of medical AI technologies. These groups will be required to maintain trust in our healthcare institutions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“a minority of AI chatbot users realized early that these LLMs do not voluntarily cite credible references to authenticate their information [10-12].”
From Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study (2024) · cites this paper as [11]
In the citing paper
“It has been argued that AI's transformational role in healthcare allows for patients to be treated as commodities and not as individuals”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Quinn et al., 2021
Citations
Cites
Principles of Biomedical Ethics
(2012) · Thieves of Virtue · doi:10.7551/mitpress/9079.003.0009 · Cited by 13,453
Shared decision making: A model for clinical practice
Elwyn, G., Frosch, D., Thomson, R., Joseph-Williams, N., Lloyd, A., Kinnersley, P., Cording, E., Tomson, D., Dodd, C., Rollnick, S., et al. (2012) · Journal of General Internal Medicine · doi:10.1007/s11606-012-2077-6 · Cited by 4,405 · cited 1×
“If practitioners fail to incorporate the values and the wishes of a specific patient into their professional decisions, the AI system may paternalistically interfere with shared decisionmaking and informed choice [18, 19].”
BackgroundPrinciples of biomedical ethics
Horton, J. (2002) · Transactions of the Royal Society of Tropical Medicine and Hygiene · doi:10.1016/s0035-9203(02)90265-8 · Cited by 4,377
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Four Models of the Physician-Patient Relationship
Emanuel, E. J. (1992) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1992.03480160079038 · Cited by 2,449
DARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360
An investigation of the Therac-25 accidents
Leveson, N. G., & Turner, C. S. (1993) · Computer · doi:10.1109/mc.1993.274940 · Cited by 1,238
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions
Evans, D. (2003) · Journal of Clinical Nursing · doi:10.1046/j.1365-2702.2003.00662.x · Cited by 993
A governance model for the application of AI in health care
Reddy, S., Allan, S., Coghlan, S., & Cooper, P. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz192 · Cited by 979
Self-citationMachine learning in medicine: Addressing ethical challenges
Vayena, E., Blasimme, A., & Cohen, I. G. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002689 · Cited by 912
Understanding adversarial attacks on deep learning based medical image analysis systems
Ma, X., Niu, Y., Gu, L., Wang, Y., Zhao, Y., Bailey, J., & Lu, F. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2020.107332 · Cited by 553
On the ethics of algorithmic decision-making in healthcare
Grote, T., & Berens, P. (2020) · Journal of Medical Ethics · doi:10.1136/medethics-2019-105586 · Cited by 551
Google DeepMind and healthcare in an age of algorithms
Powles, J., & Hodson, H. (2017) · Health and Technology · doi:10.1007/s12553-017-0179-1 · Cited by 511
The Patient as Person: Explorations in Medical Ethics.
Ralls, A., & Ramsey, P. (1972) · The Philosophical Quarterly · doi:10.2307/2217560 · Cited by 376
The Ugly Truth About Ourselves and Our Robot Creations: The Problem of Bias and Social Inequity
Howard, A., & Borenstein, J. (2018) · Science and Engineering Ethics · doi:10.1007/s11948-017-9975-2 · Cited by 337
Payrovnaziri, S. N., Chen, Z., Rengifo-Moreno, P., Miller, T., Bian, J., Chen, J. H., Liu, X., & He, Z. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa053 · Cited by 313
With an eye to AI and autonomous diagnosis
Keane, P. A., & Topol, E. J. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0048-y · Cited by 304
Artificial Intelligence and Patient-Centered Decision-Making
Bjerring, J. C., & Busch, J. (2021) · Philosophy & Technology · doi:10.1007/s13347-019-00391-6 · Cited by 274
Ethics, trust, and the professions : philosophical and cultural aspects
Pellegrino, E. D., Veatch, R. M., & Langan, J. (1991) · Cited by 272
Computer knows best? The need for value-flexibility in medical AI
McDougall, R. J. (2019) · Journal of Medical Ethics · doi:10.1136/medethics-2018-105118 · Cited by 262
From machine learning to machine reasoning: An essay
Bottou, L. (2014) · Machine Learning · doi:10.1007/s10994-013-5335-x · Cited by 221
Latent bias and the implementation of artificial intelligence in medicine
DeCamp, M., & Lindvall, C. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa094 · Cited by 198
What should medical students know about artificial intelligence in medicine?
Park, S. H., Do, K.-H., Kim, S., Park, J. H., & Lim, Y.-S. (2019) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2019.16.18 · Cited by 196
Defensive medicine: A bane to healthcare
Sekhar, M., & Vyas, N. (2013) · Annals of Medical and Health Sciences Research · doi:10.4103/2141-9248.113688 · Cited by 150
Lifting the Burden of Women's Care Work: Should Robots Replace the “Human Touch”?
Parks, J. A. (2010) · Hypatia · doi:10.1111/j.1527-2001.2009.01086.x · Cited by 110
Robots in aged care: a dystopian future?
Sparrow, R. (2016) · Ai & Society · doi:10.1007/s00146-015-0625-4 · Cited by 108
The ethics of medical AI and the physician-patient relationship
DALTON-BROWN, S. (2020) · Cambridge Quarterly of Healthcare Ethics · doi:10.1017/s0963180119000847 · Cited by 105
The right to refuse diagnostics and treatment planning by artificial intelligence
Ploug, T., & Holm, S. (2019) · Medicine Health Care and Philosophy · doi:10.1007/s11019-019-09912-8 · Cited by 96
A definition and ethical evaluation of overdiagnosis
Carter, S. M., Degeling, C., Doust, J. A., & Barratt, A. L. (2016) · Journal of Medical Ethics · doi:10.1136/medethics-2015-102928 · Cited by 91
The Computer and Clinical Judgment
Barnett, G. O. (1982) · New England Journal of Medicine · doi:10.1056/nejm198208193070808 · Cited by 57
Cited by
Artificial intelligence in healthcare: transforming the practice of medicine
Bajwa, J., Munir, U., Nori, A., & Williams, B. (2021) · Future Healthcare Journal · doi:10.7861/fhj.2021-0095 · Cited by 1,856
The impact of generative artificial intelligence on socioeconomic inequalities and policy making
Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., Bonnefon, J.-F., Brañas-Garza, P., Butera, L., Douglas, K. M., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae191 · Cited by 371
SHIFTing artificial intelligence to be responsible in healthcare: a systematic review
Siala, H., & Wang, Y. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114782 · Cited by 358
Navigating Challenges and Opportunities in Multi-Omics Integration for Personalized Healthcare
Mohr, A. E., Ortega-Santos, C. P., Whisner, C. M., Klein-Seetharaman, J., & Jasbi, P. (2024) · Biomedicines · doi:10.3390/biomedicines12071496 · Cited by 311
Artificial intelligence technology, public trust, and effective governance
Robles, P., & Mallinson, D. J. (2023) · Review of Policy Research · doi:10.1111/ropr.12555 · Cited by 235
Metadata
- Authors
- Quinn, T. P., Senadeera, M., Jacobs, S., Coghlan, S., & Le, V.
- Affiliations
- Deakin University (AU)
- The University of Melbourne (AU)
- Corresponding author
- Quinn, Thomas P
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-12-19
- Volume
- 28
- Issue
- 4
- Pages
- 890-894
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7973477
- OpenAlex
- W3117511502
- Cited by
- 287
- References
- 32
- Citations by year
- 2026: 66 · 2025: 83 · 2024: 72 · 2023: 37 · 2022: 20 · 2021: 9
- FWCI
- 4.86
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Minerva Access (University of Melbourne) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- artificial intelligence in healthcare, public trust in AI, AI failure, clinical outcomes, patient experience, professional accreditation, artificial intelligence in education
- MeSH terms
- Accreditation, Algorithms, Artificial Intelligence, Attitude to Computers, Attitude to Health, Humans, Medical Informatics, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.