Trust me, i'm a chatbot: how artificial intelligence in health care fails the Turing Test
Open accessPublished version of a preprint
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Over the next decade, one issue which will dominate sociotechnical studies in health informatics is the extent to which the promise of artificial intelligence in health care will be realized, along with the social and ethical issues which accompany it. A useful thought experiment is the application of the Turing test to user-facing artificial intelligence systems in health care (such as chatbots or conversational agents). In this paper I argue that many medical decisions require value judgements and the doctor-patient relationship requires empathy and understanding to arrive at a shared decision, often handling large areas of uncertainty and balancing competing risks. Arguably, medicine requires wisdom more than intelligence, artificial or otherwise. Artificial intelligence therefore needs to supplement rather than replace medical professionals, and identifying the complementary positioning of artificial intelligence in medical consultation is a key challenge for the future. In health care, artificial intelligence needs to pass the implementation game, not the imitation game.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“AI misuse such as using smart technology to supplant established health services has been flagged up as an ethical concern (Carter et al., 2020;Powell, 2019) that can potentially exacerbate existing health inequalities (Abràmoff et al., 2020).”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Powell, 2019
The cited source says what the claim attributes to it.
In the citing paper
“AI misuse such as using smart technology to supplant established health services has been flagged up as an ethical concern (Carter et al., 2020;Powell, 2019) that can potentially exacerbate existing health inequalities (Abràmoff et al., 2020).”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Powell, 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Studies on AI are still scarce and based on observational studies and secondary datasets.”
From 2021 ISHNE/HRS/EHRA/APHRS Expert Collaborative Statement on mHealth in Arrhythmia Management: Digital Medical Tools for Heart Rhythm Professionals: From the International Society for Holter and Noninvasive Electrocardiology/Heart Rhythm Society/European Heart Rhythm Association/Asia-Pacific Heart Rhythm Society (2021) · cites this paper as Powell 2019
Experimental — accuracy not yet measuredIn the citing paper
“Biased AI algorithms trained with unrepresentative and inequitable datasets could lead to inaccurate medical diagnosis and decisions, or worse, discriminatory profiling of citizens in low resource settings”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Powell, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769 · cited 1×
“management of individual patients [19].”
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290
Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations
GREENHALGH, T., ROBERT, G., MACFARLANE, F., BATE, P., & KYRIAKIDOU, O. (2004) · The Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00325.x · Cited by 7,448 · cited 1×
“Factors which will influence the implementation of AI tools include those related to people, such as professional and public attitudes, trust, existing work practices, training needs, and the risks of deskilling and disempowerment; those related to the health system, such as leadership and management, the positioning…”
BackgroundWhy Are There Still So Many Jobs? The History and Future of Workplace Automation
Autor, D. H. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.3 · Cited by 3,917 · cited 1×
“As various studies into the future of work have shown, automation in the workplace will not eliminate all human tasks [20].”
BackgroundYeh, M.-L. (2014) · Journal of Nursing Research · doi:10.1097/jnr.0000000000000065 · Cited by 3,829
Shared decision making — The pinnacle of patient-centered care
Barry, M. J., & Edgman-Levitan, S. (2012) · New England Journal of Medicine · doi:10.1056/nejmp1109283 · Cited by 3,433 · cited 1×
“However, we know that medical consultations are complex [17], that many medical decisions require value judgements, and that the doctor-patient relationship requires empathy and understanding to arrive at a shared decision [18].”
BackgroundGreenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887 · cited 3×
“We need to understand how to integrate the tools and practices of AI within the work and culture of professionals and organizations, to investigate factors related to adoption, nonadoption, and abandonment [10,12], and investigate the work required to sustain innovation [22].”
BackgroundEysenbach, G., Powell, J., Kuss, O., & Sa, E.-R. (2002) · JAMA · doi:10.1001/jama.287.20.2691 · Cited by 1,871 · cited 1×
“From a sociotechnical perspective, there were initial preoccupations with the impact of a new, uncontrolled form of mass communication, alongside concerns with the quality of unregulated online information and threats to professions, with medical professionals in particular fearing a loss of authority [1-3].”
BackgroundSelf-citationEysenbach, G., Powell, J., Englesakis, M., Rizo, C., & Stern, A. (2004) · BMJ · doi:10.1136/bmj.328.7449.1166 · Cited by 1,268
Self-citationMachine learning in medicine: Addressing ethical challenges
Vayena, E., Blasimme, A., & Cohen, I. G. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002689 · Cited by 912 · cited 1×
“For those of us studying the social science of digital technology in health and health care, one area of research is likely to dominate the next decade: the extent to which the promise of artificial intelligence (AI) in health care will be realized, and the social and ethical issues which accompany it [11-13].”
Background
Research works printed but not linked: 1
- Innes, AD, 2005, Br J Gen Pract, 55, 47
Cited by
Artificial intelligence for good health: a scoping review of the ethics literature.
Murphy, K., Di Ruggiero, E., Upshur, R., Willison, D. J., Malhotra, N., Cai, J. C., Malhotra, N., Lui, V., & Gibson, J. (2021) · BMC Medical Ethics · doi:10.1186/s12910-021-00577-8 · Cited by 435
SHIFTing artificial intelligence to be responsible in healthcare: a systematic review
Siala, H., & Wang, Y. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114782 · Cited by 358
Artificial intelligence technologies and compassion in healthcare: A systematic scoping review
Morrow, E., Zidaru, T., Ross, F., Mason, C., Patel, K. D., Ream, M., & Stockley, R. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.971044 · Cited by 250
“There is a strong anticipation perspective relating to concerns about role replacement (Johnston, 2018; Blease et al., 2019; Bridge and Bridge, 2019; Powell, 2019; Blease et al., 2020; Doraiswamy et al., 2020; Alrassi et al., 2021) and which parts of healthcare practice, can and should be entrusted…”
BackgroundFriðgeirsson, Þ. V., Ingason, H. Þ., Jónasson, H. I., & Jónsdóttir, H. (2021) · Sustainability · doi:10.3390/su13042345 · Cited by 210
“The famous Turing Test method of inquiry for determining whether or not a computer is capable of thinking like a human being is rarely passed by a computer [2].”
MethodResponsible AI for Digital Health: a Synthesis and a Research Agenda
Trocin, C., Mikalef, P., Papamitsiou, Z., & Conboy, K. (2023) · Information Systems Frontiers · doi:10.1007/s10796-021-10146-4 · Cited by 205
“Due to the ‘black-boxed’ nature of many algorithms, it is challenging to understand the processes followed, the data used, the rules applied, and the people involved (Guan, 2019; Powell, 2019).”
BackgroundMethod
Metadata
- Authors
- Powell, J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Powell, John
- ORCID
- Powell, John
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-10-28
- Volume
- 21
- Issue
- 10
- Pages
- e16222
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6914236
- OpenAlex
- W2980109286
- Cited by
- 115
- References
- 26
- Citations by year
- 2026: 4 · 2025: 13 · 2024: 18 · 2023: 17 · 2022: 24 · 2021: 27 · 2020: 9 · 2019: 3
- FWCI
- 4.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- other-oa
- Locations
- Journal of Medical Internet Research (publishedVersion, other-oa)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- MQ: Transforming Mental Health (MQ14PE25)
- National Institute for Health and Care Research
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Health Professions › General Health Professions › Patient-Provider Communication in Healthcare
- Keywords
- artificial intelligence in healthcare, Turing test, chatbots, physician-patient relationship, shared decision-making, clinical uncertainty, human-AI collaboration, AI ethics
- MeSH terms
- Machine Learning, Artificial Intelligence, Humans, Medical Informatics, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Trust me, i'm a chatbot: how artificial intelligence in health care fails the Turing Test. https://app.cochat.ai/papers/10.2196/16222
- The paper
- Powell, J. (2019). Trust me, i'm a chatbot: how artificial intelligence in health care fails the Turing Test. Journal of Medical Internet Research. https://doi.org/10.2196/16222
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.