Trust me, i'm a chatbot: how artificial intelligence in health care fails the Turing Test
Open accessPublished version of a preprint
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Over the next decade, one issue which will dominate sociotechnical studies in health informatics is the extent to which the promise of artificial intelligence in health care will be realized, along with the social and ethical issues which accompany it. A useful thought experiment is the application of the Turing test to user-facing artificial intelligence systems in health care (such as chatbots or conversational agents). In this paper I argue that many medical decisions require value judgements and the doctor-patient relationship requires empathy and understanding to arrive at a shared decision, often handling large areas of uncertainty and balancing competing risks. Arguably, medicine requires wisdom more than intelligence, artificial or otherwise. Artificial intelligence therefore needs to supplement rather than replace medical professionals, and identifying the complementary positioning of artificial intelligence in medical consultation is a key challenge for the future. In health care, artificial intelligence needs to pass the implementation game, not the imitation game.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“AI misuse such as using smart technology to supplant established health services has been flagged up as an ethical concern (Carter et al., 2020;Powell, 2019) that can potentially exacerbate existing health inequalities (Abràmoff et al., 2020).”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Powell, 2019
The cited source says what the claim attributes to it.
In the citing paper
“AI misuse such as using smart technology to supplant established health services has been flagged up as an ethical concern (Carter et al., 2020;Powell, 2019) that can potentially exacerbate existing health inequalities (Abràmoff et al., 2020).”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Powell, 2019
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Studies on AI are still scarce and based on observational studies and secondary datasets.”
From 2021 ISHNE/HRS/EHRA/APHRS Expert Collaborative Statement on mHealth in Arrhythmia Management: Digital Medical Tools for Heart Rhythm Professionals: From the International Society for Holter and Noninvasive Electrocardiology/Heart Rhythm Society/European Heart Rhythm Association/Asia-Pacific Heart Rhythm Society (2021) · cites this paper as Powell 2019
Experimental — accuracy not yet measuredIn the citing paper
“Biased AI algorithms trained with unrepresentative and inequitable datasets could lead to inaccurate medical diagnosis and decisions, or worse, discriminatory profiling of citizens in low resource settings”
From SHIFTing artificial intelligence to be responsible in healthcare: a systematic review (2022) · cites this paper as Powell, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769 · cited 1×
“management of individual patients [19].”
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290
Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations
GREENHALGH, T., ROBERT, G., MACFARLANE, F., BATE, P., & KYRIAKIDOU, O. (2004) · The Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00325.x · Cited by 7,448 · cited 1×
“Factors which will influence the implementation of AI tools include those related to people, such as professional and public attitudes, trust, existing work practices, training needs, and the risks of deskilling and disempowerment; those related to the health system, such as leadership and management, the positioning…”
BackgroundWhy Are There Still So Many Jobs? The History and Future of Workplace Automation
Autor, D. H. (2015) · Journal of Economic Perspectives · doi:10.1257/jep.29.3.3 · Cited by 3,917 · cited 1×
“As various studies into the future of work have shown, automation in the workplace will not eliminate all human tasks [20].”
BackgroundYeh, M.-L. (2014) · Journal of Nursing Research · doi:10.1097/jnr.0000000000000065 · Cited by 3,829
Shared decision making — The pinnacle of patient-centered care
Barry, M. J., & Edgman-Levitan, S. (2012) · New England Journal of Medicine · doi:10.1056/nejmp1109283 · Cited by 3,433 · cited 1×
“However, we know that medical consultations are complex [17], that many medical decisions require value judgements, and that the doctor-patient relationship requires empathy and understanding to arrive at a shared decision [18].”
BackgroundGreenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887 · cited 3×
“We need to understand how to integrate the tools and practices of AI within the work and culture of professionals and organizations, to investigate factors related to adoption, nonadoption, and abandonment [10,12], and investigate the work required to sustain innovation [22].”
BackgroundEysenbach, G., Powell, J., Kuss, O., & Sa, E.-R. (2002) · JAMA · doi:10.1001/jama.287.20.2691 · Cited by 1,871 · cited 1×
“From a sociotechnical perspective, there were initial preoccupations with the impact of a new, uncontrolled form of mass communication, alongside concerns with the quality of unregulated online information and threats to professions, with medical professionals in particular fearing a loss of authority [1-3].”
BackgroundSelf-citationEysenbach, G., Powell, J., Englesakis, M., Rizo, C., & Stern, A. (2004) · BMJ · doi:10.1136/bmj.328.7449.1166 · Cited by 1,268
Self-citationMachine learning in medicine: Addressing ethical challenges
Vayena, E., Blasimme, A., & Cohen, I. G. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002689 · Cited by 912 · cited 1×
“For those of us studying the social science of digital technology in health and health care, one area of research is likely to dominate the next decade: the extent to which the promise of artificial intelligence (AI) in health care will be realized, and the social and ethical issues which accompany it [11-13].”
BackgroundDoctor in the house: the Internet as a source of lay health knowledge and the challenge to expertise
Hardey, M. (1999) · Sociology of Health & Illness · doi:10.1111/1467-9566.00185 · Cited by 519
Artificial intelligence and the implementation challenge
Shaw, J., Rudzicz, F., Jamieson, T., & Goldfarb, A. (2019) · Journal of Medical Internet Research · doi:10.2196/13659 · Cited by 389 · cited 4×
“Factors which will influence the implementation of AI tools include those related to people, such as professional and public attitudes, trust, existing work practices, training needs, and the risks of deskilling and disempowerment; those related to the health system, such as leadership and management, the positioning…”
BackgroundInfluentialPhysicians’ perceptions of chatbots in health Care: cross-sectional web-based survey
Palanica, A., Flaschner, P., Thommandram, A., Li, M., & Fossat, Y. (2019) · Journal of Medical Internet Research · doi:10.2196/12887 · Cited by 384 · cited 2×
“applications and intelligent agents which interact with people in real-time, using inferences to provide advice or instruction based on probabilities which the tool can derive and improve over time, such as a chatbot substituting or complementing a health care consultation with a patient [15].”
BackgroundArtificial Intelligence Versus Clinicians in Disease Diagnosis: Systematic Review
Shen, J., Zhang, C. J. P., Jiang, B., Chen, J., Song, J., Liu, Z., He, Z., Wong, S. Y., Fang, P.-H., & Ming, W.-K. (2019) · JMIR Medical Informatics · doi:10.2196/10010 · Cited by 373 · cited 1×
“Firstly, there are data-facing applications which use techniques such as machine learning and artificial neural networks to derive new knowledge from large datasets, such as improving diagnostic accuracy from scans and other images [14].”
MethodCresswell, K. M., Bates, D. W., & Sheikh, A. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001684 · Cited by 364 · cited 2×
“Factors which will influence the implementation of AI tools include those related to people, such as professional and public attitudes, trust, existing work practices, training needs, and the risks of deskilling and disempowerment; those related to the health system, such as leadership and management, the positioning…”
BackgroundComputing Machinery and Intelligence
Turing, A. M. (2004) · The Turing Test · doi:10.7551/mitpress/6928.003.0012 · Cited by 273
The importance of being expert: the quest for cancer information on the Internet
Ziebland, S. (2004) · Social Science & Medicine · doi:10.1016/j.socscimed.2004.02.019 · Cited by 225 · cited 1×
“From a sociotechnical perspective, there were initial preoccupations with the impact of a new, uncontrolled form of mass communication, alongside concerns with the quality of unregulated online information and threats to professions, with medical professionals in particular fearing a loss of authority [1-3].”
BackgroundHardey, M. (2001) · Information, Communication & Society · doi:10.1080/713768551 · Cited by 217 · cited 1×
“0 developments took hold and the public became producers as well as consumers of health information, researchers began to identify the benefits of online peer-to-peer communication and the sharing of information in virtual communities, social media, and increasingly on health ratings sites [4-7].”
BackgroundRathbone, A. L., Clarry, L., & Prescott, J. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8598 · Cited by 158 · cited 1×
“With the mass uptake in smartphones, the subsequent rapid developments in mobile health, and the explosion in health apps, we are now exploring the value of low-cost, patient-centered interventions delivered directly to consumers [8,9].”
BackgroundPowell, J., Hamborg, T., Stallard, N., Burls, A., McSorley, J., Bennett, K., Griffiths, K. M., & Christensen, H. (2012) · Journal of Medical Internet Research · doi:10.2196/jmir.2240 · Cited by 156 · cited 1×
“With the mass uptake in smartphones, the subsequent rapid developments in mobile health, and the explosion in health apps, we are now exploring the value of low-cost, patient-centered interventions delivered directly to consumers [8,9].”
BackgroundSelf-citationComplex consultations and the 'edge of chaos'.
Innes, A., Campion, P. D., & Griffiths, F. E. (2005) · Europe PMC (PubMed Central) · Cited by 126
Pope, C., Halford, S., Turnbull, J., Prichard, J., Calestani, M., & May, C. (2013) · BMC Health Services Research · doi:10.1186/1472-6963-13-111 · Cited by 101 · cited 1×
“We need to understand how to integrate the tools and practices of AI within the work and culture of professionals and organizations, to investigate factors related to adoption, nonadoption, and abandonment [10,12], and investigate the work required to sustain innovation [22].”
BackgroundCross-sectional survey of users of Internet depression communities
Powell, J., McCarthy, N., & Eysenbach, G. (2003) · BMC Psychiatry · doi:10.1186/1471-244x-3-19 · Cited by 96 · cited 1×
“0 developments took hold and the public became producers as well as consumers of health information, researchers began to identify the benefits of online peer-to-peer communication and the sharing of information in virtual communities, social media, and increasingly on health ratings sites [4-7].”
BackgroundSelf-citationvan Velthoven, M. H., Atherton, H., & powell, J. A. (2018) · Patient Education and Counseling · doi:10.1016/j.pec.2018.04.001 · Cited by 36
Self-citationA study of automated self-assessment in a primary care student health centre setting
Poote, A., French, D., Dale, J. R., & powell, J. A. (2014) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x14529246 · Cited by 27 · cited 1×
“To take an example, in an early study of an algorithm-based triage tool in primary care, we showed that physicians lacked trust in the ability of the machine to take clinical risks and worried about issues of governance and accountability, such that the sensitivity of the tool, in terms of the urgency of triage, was c…”
MethodSelf-citationRosly, H. M., & Rosly, M. M. (2019) · CRC Press eBooks · doi:10.1201/9781351032186-18 · Cited by 1
Research works printed but not linked: 1
- Innes, AD, 2005, Br J Gen Pract, 55, 47
Cited by
Artificial intelligence for good health: a scoping review of the ethics literature.
Murphy, K., Di Ruggiero, E., Upshur, R., Willison, D. J., Malhotra, N., Cai, J. C., Malhotra, N., Lui, V., & Gibson, J. (2021) · BMC Medical Ethics · doi:10.1186/s12910-021-00577-8 · Cited by 435
SHIFTing artificial intelligence to be responsible in healthcare: a systematic review
Siala, H., & Wang, Y. (2022) · Social Science & Medicine · doi:10.1016/j.socscimed.2022.114782 · Cited by 358
Artificial intelligence technologies and compassion in healthcare: A systematic scoping review
Morrow, E., Zidaru, T., Ross, F., Mason, C., Patel, K. D., Ream, M., & Stockley, R. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.971044 · Cited by 250
“There is a strong anticipation perspective relating to concerns about role replacement (Johnston, 2018; Blease et al., 2019; Bridge and Bridge, 2019; Powell, 2019; Blease et al., 2020; Doraiswamy et al., 2020; Alrassi et al., 2021) and which parts of healthcare practice, can and should be entrusted…”
BackgroundFriðgeirsson, Þ. V., Ingason, H. Þ., Jónasson, H. I., & Jónsdóttir, H. (2021) · Sustainability · doi:10.3390/su13042345 · Cited by 210
“The famous Turing Test method of inquiry for determining whether or not a computer is capable of thinking like a human being is rarely passed by a computer [2].”
MethodResponsible AI for Digital Health: a Synthesis and a Research Agenda
Trocin, C., Mikalef, P., Papamitsiou, Z., & Conboy, K. (2023) · Information Systems Frontiers · doi:10.1007/s10796-021-10146-4 · Cited by 205
“Due to the ‘black-boxed’ nature of many algorithms, it is challenging to understand the processes followed, the data used, the rules applied, and the people involved (Guan, 2019; Powell, 2019).”
BackgroundMethod
Metadata
- Authors
- Powell, J.
- Affiliations
- University of Oxford (GB)
- Corresponding author
- Powell, John
- ORCID
- Powell, John
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2019-10-28
- Volume
- 21
- Issue
- 10
- Pages
- e16222
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC6914236
- OpenAlex
- W2980109286
- Cited by
- 115
- References
- 26
- Citations by year
- 2026: 4 · 2025: 13 · 2024: 18 · 2023: 17 · 2022: 24 · 2021: 27 · 2020: 9 · 2019: 3
- FWCI
- 4.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- other-oa
- Locations
- Journal of Medical Internet Research (publishedVersion, other-oa)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- MQ: Transforming Mental Health (MQ14PE25)
- National Institute for Health and Care Research
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Health Sciences › Health Professions › General Health Professions › Patient-Provider Communication in Healthcare
- Keywords
- artificial intelligence in healthcare, Turing test, chatbots, physician-patient relationship, shared decision-making, clinical uncertainty, human-AI collaboration, AI ethics
- MeSH terms
- Machine Learning, Artificial Intelligence, Humans, Medical Informatics, Telemedicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.