Trust and medical AI: the challenges we face and the expertise needed to overcome them
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) is increasingly of tremendous interest in the medical field. How-ever, failures of medical AI could have serious consequences for both clinical outcomes and the patient experience. These consequences could erode public trust in AI, which could in turn undermine trust in our healthcare institutions. This article makes 2 contributions. First, it describes the major conceptual, technical, and humanistic challenges in medical AI. Second, it proposes a solution that hinges on the education and accreditation of new expert groups who specialize in the development, verification, and operation of medical AI technologies. These groups will be required to maintain trust in our healthcare institutions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“a minority of AI chatbot users realized early that these LLMs do not voluntarily cite credible references to authenticate their information [10-12].”
From Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study (2024) · cites this paper as [11]
In the citing paper
“It has been argued that AI's transformational role in healthcare allows for patients to be treated as commodities and not as individuals”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Quinn et al., 2021
Citations
Cites
Principles of Biomedical Ethics
(2012) · Thieves of Virtue · doi:10.7551/mitpress/9079.003.0009 · Cited by 13,453
Shared decision making: A model for clinical practice
Elwyn, G., Frosch, D., Thomson, R., Joseph-Williams, N., Lloyd, A., Kinnersley, P., Cording, E., Tomson, D., Dodd, C., Rollnick, S., et al. (2012) · Journal of General Internal Medicine · doi:10.1007/s11606-012-2077-6 · Cited by 4,405 · cited 1×
“If practitioners fail to incorporate the values and the wishes of a specific patient into their professional decisions, the AI system may paternalistically interfere with shared decisionmaking and informed choice [18, 19].”
BackgroundPrinciples of biomedical ethics
Horton, J. (2002) · Transactions of the Royal Society of Tropical Medicine and Hygiene · doi:10.1016/s0035-9203(02)90265-8 · Cited by 4,377
Key challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
Four Models of the Physician-Patient Relationship
Emanuel, E. J. (1992) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1992.03480160079038 · Cited by 2,449
DARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360
An investigation of the Therac-25 accidents
Leveson, N. G., & Turner, C. S. (1993) · Computer · doi:10.1109/mc.1993.274940 · Cited by 1,238
Automation bias: a systematic review of frequency, effect mediators, and mitigators
Goddard, K., Roudsari, A., & Wyatt, J. C. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000089 · Cited by 1,162
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions
Evans, D. (2003) · Journal of Clinical Nursing · doi:10.1046/j.1365-2702.2003.00662.x · Cited by 993
Cited by
Artificial intelligence in healthcare: transforming the practice of medicine
Bajwa, J., Munir, U., Nori, A., & Williams, B. (2021) · Future Healthcare Journal · doi:10.7861/fhj.2021-0095 · Cited by 1,856
The impact of generative artificial intelligence on socioeconomic inequalities and policy making
Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., Bonnefon, J.-F., Brañas-Garza, P., Butera, L., Douglas, K. M., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae191 · Cited by 371
SHIFTing artificial intelligence to be responsible in healthcare: a systematic review
Siala, H., & Wang, Y. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114782 · Cited by 358
Navigating Challenges and Opportunities in Multi-Omics Integration for Personalized Healthcare
Mohr, A. E., Ortega-Santos, C. P., Whisner, C. M., Klein-Seetharaman, J., & Jasbi, P. (2024) · Biomedicines · doi:10.3390/biomedicines12071496 · Cited by 311
Artificial intelligence technology, public trust, and effective governance
Robles, P., & Mallinson, D. J. (2023) · Review of Policy Research · doi:10.1111/ropr.12555 · Cited by 233
Metadata
- Authors
- Quinn, T. P., Senadeera, M., Jacobs, S., Coghlan, S., & Le, V.
- Affiliations
- Deakin University (AU)
- The University of Melbourne (AU)
- Corresponding author
- Quinn, Thomas P
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-12-19
- Volume
- 28
- Issue
- 4
- Pages
- 890-894
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC7973477
- OpenAlex
- W3117511502
- Cited by
- 287
- References
- 32
- Citations by year
- 2026: 66 · 2025: 83 · 2024: 72 · 2023: 37 · 2022: 20 · 2021: 9
- FWCI
- 4.86
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Minerva Access (University of Melbourne) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- artificial intelligence in healthcare, public trust in AI, AI failure, clinical outcomes, patient experience, professional accreditation, artificial intelligence in education
- MeSH terms
- Accreditation, Algorithms, Artificial Intelligence, Attitude to Computers, Attitude to Health, Humans, Medical Informatics, Trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Trust and medical AI: the challenges we face and the expertise needed to overcome them. https://app.cochat.ai/papers/10.1093/jamia/ocaa268
- The paper
- Quinn, T. P., Senadeera, M., Jacobs, S., Coghlan, S., & Le, V. (2021). Trust and medical AI: the challenges we face and the expertise needed to overcome them. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocaa268
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.