Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study
Open accessAbstract
OBJECTIVE: The objective of the current study was to assess the quality of large language model (LLM) chatbot versus physician-generated responses to patient-generated rheumatology questions. METHODS: We conducted a single-center cross-sectional survey of rheumatology patients (n = 17) in Edmonton, Alberta, Canada. Patients evaluated LLM chatbot versus physician-generated responses for comprehensiveness and readability, with four rheumatologists also evaluating accuracy by using a Likert scale from 1 to 10 (1 being poor, 10 being excellent). RESULTS: Patients rated no significant difference between artificial intelligence (AI) and physician-generated responses in comprehensiveness (mean 7.12 ± SD 0.99 vs 7.52 ± 1.16; P = 0.1962) or readability (7.90 ± 0.90 vs 7.80 ± 0.75; P = 0.5905). Rheumatologists rated AI responses significantly poorer than physician responses on comprehensiveness (AI 5.52 ± 2.13 vs physician 8.76 ± 1.07; P < 0.0001), readability (AI 7.85 ± 0.92 vs physician 8.75 ± 0.57; P = 0.0003), and accuracy (AI 6.48 ± 2.07 vs physician 9.08 ± 0.64; P < 0.0001). The proportion of preference to AI- versus physician-generated responses by patients and physicians was 0.45 ± 0.18 and 0.15 ± 0.08, respectively (P = 0.0106). After learning that one answer for each question was AI generated, patients were able to correctly identify AI-generated answers at a lower proportion compared to physicians (0.49 ± 0.26 vs 0.97 ± 0.04; P = 0.0183). The average word count of AI answers was 69.10 ± 25.35 words, as compared to 98.83 ± 34.58 words for physician-generated responses (P = 0.0008). CONCLUSION: Rheumatology patients rated AI-generated responses to patient questions similarly to physician-generated responses in terms of comprehensiveness, readability, and overall preference. However, rheumatologists rated AI responses significantly poorer than physician-generated responses, suggesting that LLM chatbot responses are inferior to physician responses, a difference that patients may not be aware of.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Currently, CHAS are inconsistent in their design, analysis, and reporting.
“Currently, these studies are inconsistent in their design, analysis, and reporting.14,15,16”
From Large Language Models for Chatbot Health Advice Studies: A Systematic Review (2025) · cites this paper as [16]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review
Milne-Ives, M., de Cock, C., Lim, E., Shehadeh, M. H., de Pennington, N., Mole, G., Normando, E., & Meinert, E. (2020) · Journal of Medical Internet Research · doi:10.2196/20346 · Cited by 626 · see citing works
Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., Srinivasan, N., Park, J., Burch, M., Watson, R., et al. (2023) · Obesity Surgery · doi:10.1007/s11695-023-06603-5 · Cited by 282 · see citing works
Assessing the impact of the COVID-19 pandemic on clinician ambulatory electronic health record use
Holmgren, A. J., Downing, N. L., Tang, M., Sharp, C., Longhurst, C., & Huckman, R. S. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab268 · Cited by 182 · see citing works
Sng, G. G. R., Tung, J. Y. M., Lim, D. Y. Z., & Bee, Y. M. (2023) · Diabetes Care · doi:10.2337/dc23-0197 · Cited by 160 · see citing works
Zhu, L., Mou, W., & Chen, R. (2023) · Journal of Translational Medicine · doi:10.1186/s12967-023-04123-5 · Cited by 144 · see citing works
On the cusp: considering the impact of artificial intelligence language models in healthcare
Goodman, R. S., Patrinely, J. R., Osterman, T., Wheless, L., & Johnson, D. B. (2023) · Med · doi:10.1016/j.medj.2023.02.008 · Cited by 100 · see citing works
Lahat, A., Shachar, E., Avidan, B., Glicksberg, B. S., & Klang, E. (2023) · Diagnostics · doi:10.3390/diagnostics13111950 · Cited by 92 · see citing works
Rasmussen, M. L. R., Larsen, A., Subhi, Y., & Potapenko, I. O. (2023) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-023-06078-1 · Cited by 76 · see citing works
Association Between Billing Patient Portal Messages as e-Visits and Patient Messaging Volume
Holmgren, A. J., Byron, M. E., Grouse, C. K., & Adler‐Milstein, J. (2023) · JAMA · doi:10.1001/jama.2022.24710 · Cited by 74 · see citing works
The Electronic Health Record Inbox: Recommendations for Relief
Sinsky, C. A., Shanafelt, T. D., & Ripp, J. A. (2022) · Journal of General Internal Medicine · doi:10.1007/s11606-022-07766-0 · Cited by 49 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Current applications and challenges in large language models for patient care: a systematic review
Busch, F., Hoffmann, L., Rueger, C., van Dijk, E. H., Kader, R., Ortiz-Prado, E., Makowski, M. R., Saba, L., Hadamitzky, M., Kather, J. N., et al. (2025) · Communications Medicine · doi:10.1038/s43856-024-00717-2 · Cited by 358 · see citing works
Large language models in patient education: a scoping review of applications in medicine
Aydin, S., Karabacak, M., Vlachos, V., & Margetis, K. D. (2024) · Frontiers in Medicine · doi:10.3389/fmed.2024.1477898 · Cited by 333 · see citing works
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Roles and potential of Large language models in healthcare: A comprehensive review
Lin, C., & Kuo, C. (2025) · Biomedical Journal · doi:10.1016/j.bj.2025.100868 · Cited by 102 · see citing works
Evaluating large language models and agents in healthcare: key challenges in clinical applications
Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., & Shi, D. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.03.002 · Cited by 87 · see citing works
Development and Evaluation of a Retrieval-Augmented Large Language Model Framework for Ophthalmology
Luo, M.-J., Pang, J., Bi, S., Lai, Y., Zhao, J., Shang, Y., Cui, T., Yang, Y., Lin, Z., Zhao, L., et al. (2024) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2024.2513 · Cited by 70 · see citing works
Evaluation and mitigation of cognitive biases in medical language models
Schmidgall, S., Harris, C., Essien, I., Olshvang, D., Rahman, T., Kim, J. W., Ziaei, R., Eshraghian, J., Abadir, P., & Chellappa, R. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01283-6 · Cited by 68 · see citing works
Kara, M., Özduran, E., Kara, M. M., Özbek, İ. C., & Hancı, V. (2025) · PLoS ONE · doi:10.1371/journal.pone.0326351 · Cited by 61 · see citing works
Omar, M., Naffaa, M. E., Glicksberg, B. S., Reuveni, H., Nadkarni, G. N., & Klang, E. (2024) · Rheumatology Advances in Practice · doi:10.1093/rap/rkae120 · Cited by 23 · see citing works
Xu, D., Zhao, J., Liu, R., Dai, Y., Sun, K., Wong, P. C. H., Ming, S. L. S., Wearn, K. L., Wang, J., Xie, S., et al. (2024) · Lara D. Veeken · doi:10.1093/rheumatology/keae238 · Cited by 19 · see citing works
RAGing ahead in rheumatology: new language model architectures to tame artificial intelligence
Benavent, D., Venerito, V., & Vegas, X. M. (2025) · Therapeutic Advances in Musculoskeletal Disease · doi:10.1177/1759720x251331529 · Cited by 19 · see citing works
Ruben, M. A., Blanch‐Hartigan, D., & Hall, J. A. (2025) · Journal of General Internal Medicine · doi:10.1007/s11606-025-10068-w · Cited by 14 · see citing works
Pavon, J. M., Schlientz, D., Maciejewski, M. L., Economou-Zavlanos, N., & Lee, R. (2025) · Diabetes Care · doi:10.2337/dci24-0079 · Cited by 12 · see citing works
Allen, J. W., Schaefer, G. O., Mann, S. P., Earp, B. D., & Wilkinson, D. J. C. (2024) · Research Ethics · doi:10.1177/17470161241298726 · Cited by 11 · see citing works
Rao, A. S., Mu, A., Enichen, E. J., Gupta, D., Hall, N., Koranteng, E., Marks, W., Senter-Zapata, M. J., Whitehead, D. C., White, B. A., et al. (2025) · Annals of Biomedical Engineering · doi:10.1007/s10439-025-03701-6 · Cited by 12 · see citing works
Allen, J. W., Levy, N. L., & Wilkinson, D. J. C. (2025) · Bioethics · doi:10.1111/bioe.70030 · Cited by 10 · see citing works
Allen, J. W., Wilkinson, D. J. C., & Savulescu, J. (2025) · Bioethics · doi:10.1111/bioe.70032 · Cited by 11 · see citing works
Vladić, N., Nopp, S., Pabinger, I., Ageno, W., Connors, J. M., Eichinger, S., & Ay, C. (2025) · Journal of Thrombosis and Haemostasis · doi:10.1016/j.jtha.2025.09.004 · Cited by 10 · see citing works
Farook, T. H., & Dudley, J. (2024) · Clinical and Experimental Dental Research · doi:10.1002/cre2.70028 · Cited by 10 · see citing works
Li, Y., Li, Z., Li, J., Liu, L., Liu, Y., Zhu, B., Shi, K., Lu, Y., Li, Y., Zeng, X., et al. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.105961 · Cited by 10 · see citing works
Profiling of Cardiogenic Shock: Incorporating Machine Learning Into Bedside Management
Zweck, E., Li, S., Burkhoff, D., & Kapur, N. K. (2024) · Journal of the Society for Cardiovascular Angiography & Interventions · doi:10.1016/j.jscai.2024.102047 · Cited by 9 · see citing works
Self-citationComparing the accuracy of large language models and prompt engineering in diagnosing realworld cases
Yao, G., Zhang, W. D., Zhu, Y., Wong, U.-K., Zhang, Y., Cui, Y., Shen, G., Li, Z., & Gao, H. Y. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106026 · Cited by 10 · see citing works
Jayasrilakshmi, S., & Mahapatra, A. (2026) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2026.111469 · Cited by 12 · see citing works
Wang, E., Smith, J. D., Katz, S. J., Bishay, M., Dissanayake, T. D., Jones, N., Reddy, S., SHOLTER, D. E., Soo, J., & Ye, C. (2025) · Rheumatology Advances in Practice · doi:10.1093/rap/rkaf083 · Cited by 5 · see citing works
Self-citationErden, Y., Temel, M. H., & Bağcıer, F. (2025) · Reumatologia/Rheumatology · doi:10.5114/reum/207526 · Cited by 6 · see citing works
Artificial intelligence for autoimmune diseases
Mahajan, A., LaChance, A., Rodman, A. M., Chung, L. S., & Powell, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02015-0 · Cited by 4 · see citing works
Bashir, M., El–Sappagh, S., Woo, S. S., Kim, D. I., & Abuhmed, T. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104507 · Cited by 5 · see citing works
Wang, G., Yang, R., Zhang, Y., Wen, X., Liu, C., Liu, E., Tang, M., Xue, L., & Liu, Z. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106172 · Cited by 5 · see citing works
Robinson, M. A., Belzberg, M., Cai, C., Lim, J. S., Liu, T. Y. A., & Ng, E. (2025) · JAAD International · doi:10.1016/j.jdin.2025.04.005 · Cited by 4 · see citing works
Duan, L., Yao, Z., Li, X., Wu, Y., & Sheng, D. (2025) · Digital Health · doi:10.1177/20552076251388140 · Cited by 3 · see citing works
Comparing Performance of Large Language Model-Based Tools on Patient-Driven Glaucoma Inquiries
Gupta, D., Wagner, S. L., Castillejos, A. G., Gross, A. W., Lu, E. S., Chang, E. K., Rao, A. S., & Succi, M. D. (2025) · Journal of Glaucoma · doi:10.1097/ijg.0000000000002627 · Cited by 2 · see citing works
Assessment of AI-generated responses to common pregnancy questions: A blinded expert evaluation
Atay, A. O., Atay, F., & Corlu, O. (2026) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2026.114960 · Cited by 2 · see citing works
Felde, S., Buchkremer, R., Chehab, G., Thielscher, C., Distler, J. H. W., Schneider, M., & Richter, J. G. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1817215 · Cited by 2 · see citing works
Messelink, M. A., Fadaei, S., Verhoef, L. M., Welsing, P. M. J., Nijhof, N., & Westland, H. (2024) · Rheumatology · doi:10.1093/rheumatology/keae202 · Cited by 2 · see citing works
Comparative Analysis of Large Language Models for Answering Cancer-Related Questions in Korean
Chang, H. K., Jung, J.-W., & Kim, Y. (2025) · Yonsei Medical Journal · doi:10.3349/ymj.2024.0200 · Cited by 2 · see citing works
Tamayo, L. P. V., Nishiyama, T., Peng, S., Wakamiya, S., & Aramaki, E. (2026) · JMIR Medical Informatics · doi:10.2196/80416 · Cited by 1 · see citing works
Strachna, O., & Asan, O. (2025) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/10711813251370743 · Cited by 1 · see citing works
Brooks, J. S., Blankson, P., Campbell, P. M., Cowley, R. A., Yang, T. Y., Oseni, T. I. A., Rodriguez, A., & Idris, M. Y. (2025) · JMIR Formative Research · doi:10.2196/82487 · Cited by 1 · see citing works
Venerito, V., Lopalco, G., Iannone, F., & Gupta, L. (2026) · The Lancet Rheumatology · doi:10.1016/s2665-9913(26)00076-7 · Cited by 1 · see citing works
Cangül, S., Taşyürek, M., Tunç, T., Adıgüzel, Ö., & Ortaç, H. (2026) · BMC Oral Health · doi:10.1186/s12903-026-08255-0 · Cited by 1 · see citing works
Tvrdá, L., Burton, J. K., McConnell, K., Mavromati, K., Knoche, H. O., Mikulík, R., & Quinn, T. J. (2026) · European Stroke Journal · doi:10.1093/esj/aakag037 · Cited by 1 · see citing works
Alati, A., Pigné, G., Brugère, C.-A., & Bibault, J. (2026) · JMIR Medical Education · doi:10.2196/89152 · Cited by 1 · see citing works
Daungsupawong, H., & Wiwanitkit, V. (2023) · Arthritis & Rheumatology · doi:10.1002/art.42773
Ye, C., & Ma, Z. (2023) · Arthritis & Rheumatology · doi:10.1002/art.42771
Self-citationYe, C. (2024) · Arthritis & Rheumatology · doi:10.1002/art.42796
Self-citationWang, G., Zhuo, N., & Liu, Z. (2024) · Arthritis & Rheumatology · doi:10.1002/art.42799
Izadi, A., Mosavari, H., Hosseininasab, A., Jaliliyan, A., Jafari, A., Akhlaghpasand, M., Rostami, A., Moradi‐Lakeh, M., & Eghbali, F. (2024) · doi:10.2196/preprints.67101
Honda, S., Ikari, K., Fujisaki, M., Tanaka, E., & Harigai, M. (2024) · medRxiv · doi:10.1101/2024.10.14.24315448
ChatGPT versus Radiology Institutional Websites: What Is the Patients' Point of View?
Crimì, F., Sussan, G., & D’Alessandro, C. (2025) · Radiology · doi:10.1148/radiol.241941
Gao, H., Zhang, W., Liu, S., Li, Y., Zhu, Y., Long, T., Yu, R., Guo, Q., Zou, Y., Li, J., et al. (2024) · JMIR AI · doi:10.2196/64700
Lin, S. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-5837-5.ch008
Vladić, N., Nopp, S., Pabinger, I., Ageno, W., Connors, J. M., Eichinger, S., & Ay, C. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5255286
Du, H., Li, J., Xiao, B., Wang, X., Xue, W., Deng, M., Tan, S., Wang, J., Luo, C., Chen, B., et al. (2025) · Research Square · doi:10.21203/rs.3.rs-7066667/v1
Honda, S., Ikari, K., Fujisaki, M., Tanaka, E., & Harigai, M. (2026) · Modern Rheumatology · doi:10.1093/mr/roag039
Brooks, J., Blankson, P., Campbell, P. M., Cowley, R. A., Yang, T. Y., Oseni, T. I. A., Rodriguez, A., & Idris, M. Y. (2025) · medRxiv · doi:10.1101/2025.08.15.25333788
GLM-DM: language model boosted neural networks for HbA1c trend prediction in diabetes mellitus
Ban, Y., He, X., Verona, P. M., Cook, C. B., & He, J. (2025) · Future Science OA · doi:10.1080/20565623.2025.2567166
Alati, A., Pigné, G., Brugère, C.-A., & Bibault, J. (2025) · doi:10.2196/preprints.89152
Izadi, A., Mosavari, H., Hosseininasab, A., Jaliliyan, A., Jafari, A., Akhlaghpasand, M., Rostami, A., Moradi‐Lakeh, M., & Eghbali, F. (2026) · Journal of Obesity · doi:10.1155/jobe/2376530 · Cited by 1 · see citing works
IA en reumatología: Aplicaciones clínicas actuales y consideraciones prácticas
Chacón, M., Hidalgo-Esparza, A. A., Fernández-Manrique, M. D. L. Á., Bustos, I. G., Cedeño, M. M. F., Figueroa, G. R. S., & Vargas-Morales, D. (2026) · Revista Reumatología al Día. · doi:10.62484/rad.v20i1.12402
Demystifying Artificial Intelligence
Collins, J. E., Harari, R. E., & Duryea, J. (2026) · Rheumatic Disease Clinics of North America · doi:10.1016/j.rdc.2026.04.001
When the algorithm disagrees: managing AI-generated ‘second opinions’ in clinical neurophysiology
Maitland, S., & Baker, M. R. (2026) · Clinical Neurophysiology · doi:10.1016/j.clinph.2026.2112325
Song, Y., Mao, Y., Song, S., Li, S., & Zhang, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-9988294/v1
Chen, R., Pan, Y., Liang, M., Wang, A., Ma, Z., Xi, Y., Fan, N., Yuan, S., Du, P., Wang, T., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/87816
Self-citationBernstein, E. M., Bol, B., Shanaa, J., Ramsamooj, A., Baldini, T. D., & Lum, Z. C. (2026) · Cureus · doi:10.7759/cureus.113654
Patient-perceived quality of artificial intelligence responses in Hashimoto's thyroiditis
Aydın, R. F., Aksu, Ö. B., & Güllü, S. (2026) · Revista da Associação Médica Brasileira · doi:10.1590/1806-9282.20260144
Xu, T., Yin, Y., Zhang, X., Wei, W., Yuan, J., Wang, H., Ding, G., Xue, W., Chen, Z., Xie, S., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/98118
Arif, M., Wright, C., Rodriguez, S., Graves, S., Krishnamurti, T., & Talabi, M. B. (2026) · Arthritis Care & Research · doi:10.1002/acr.80168
Multidimensional linguistic evaluation of LLM-generated and resident-rewritten discharge summaries
Trindade, R. P. D. A., Senra, L. P. C., Feitosa, G. F., & dos Santos, E. R. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106770
Kremer, P., Haase, I., Boyadzhieva, Z., & Krusche, M. (2026) · RMD Open · doi:10.1136/rmdopen-2026-007538
- Authors
- Ye, C., Zweck, E., Ma, Z., Smith, J., & Katz, S.
- Affiliations
- University of Alberta (CA)
- Düsseldorf University Hospital (DE)
- Corresponding author
- Ye, Carrie
- Venue
- Arthritis & Rheumatology
- Publisher
- Wiley
- Published
- 2024-01-18
- Volume
- 76
- Issue
- 3
- Pages
- 479-484
- ISSN
- 2326-5191 (ISSN-L)
- 2326-5205
- OpenAlex
- W4388014118
- Cited by
- 69
- References
- 11
- Citations by year
- 2026: 22 · 2025: 31 · 2024: 14 · 2023: 2
- FWCI
- 2.76
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Arthritis & Rheumatology (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › General Health Professions › Health Literacy and Information Accessibility
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- large language models, AI chatbots, patient satisfaction, readability, response accuracy, patient-physician communication, artificial intelligence in healthcare, rheumatology patients
- MeSH terms
- Alberta, Artificial Intelligence, Cross-Sectional Studies, Humans, Language, Physicians, Rheumatology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
