Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4
Open accessAbstract
Pre-clinical studies suggest that large language models (i.e., ChatGPT) could be used in the diagnostic process to distinguish inflammatory rheumatic (IRD) from other diseases. We therefore aimed to assess the diagnostic accuracy of ChatGPT-4 in comparison to rheumatologists. For the analysis, the data set of Gräf et al. (2022) was used. Previous patient assessments were analyzed using ChatGPT-4 and compared to rheumatologists’ assessments. ChatGPT-4 listed the correct diagnosis comparable often to rheumatologists as the top diagnosis 35% vs 39% ( p = 0.30); as well as among the top 3 diagnoses, 60% vs 55%, ( p = 0.38). In IRD-positive cases, ChatGPT-4 provided the top diagnosis in 71% vs 62% in the rheumatologists’ analysis. Correct diagnosis was among the top 3 in 86% (ChatGPT-4) vs 74% (rheumatologists). In non-IRD cases, ChatGPT-4 provided the correct top diagnosis in 15% vs 27% in the rheumatologists’ analysis. Correct diagnosis was among the top 3 in non-IRD cases in 46% of the ChatGPT-4 group vs 45% in the rheumatologists group. If only the first suggestion for diagnosis was considered, ChatGPT-4 correctly classified 58% of cases as IRD compared to 56% of the rheumatologists ( p = 0.52). ChatGPT-4 showed a slightly higher accuracy for the top 3 overall diagnoses compared to rheumatologist’s assessment. ChatGPT-4 was able to provide the correct differential diagnosis in a relevant number of cases and achieved better sensitivity to detect IRDs than rheumatologist, at the cost of lower specificity. The pilot results highlight the potential of this new technology as a triage tool for the diagnosis of IRD.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
General-purpose LLMs such as ChatGPT can achieve diagnostic accuracy comparable to that of experienced rheumatologists.
“Krusche et al. demonstrated in 2023 that general-purpose LLMs, such as ChatGPT, can achieve diagnostic accuracy comparable to that of experienced rheumatologists [10].”
From Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation (2026) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · see citing works
Flagged first, then by citations
Research works printed but not linked: 14
- [1]Rheumadocs und Arbeitskreis Junge Rheumatologie (AGJR), Krusche M, Sewerin P, Kleyer A, Mucke J, Vossen D, u. a. Facharztweiterbildung quo vadis? Z Für Rheumatol. Oktober 2019;78(8):692–7. 31468164
- [2]Miloslavsky EM Marston B The challenge of addressing the rheumatology workforce shortage J Rheumatol Juni 2022 49 6 555 557 10.3899/jrheum.220300 35428725 · DOI
- [3]Fuchs F Morf H Mohn J Mühlensiepen F Ignatyev Y Bohr D Diagnostic delay stages and pre-diagnostic treatment in patients with suspected rheumatic diseases before special care consultation: results of a multicenter-based study Rheumatol Int März 2023 43 3 495 502 10.1007/s00296-022-05223-z PMC9968271 36214864 · DOI
- [4]Knitza J Mohn J Bergmann C Kampylafka E Hagen M Bohr D Accuracy, patient-perceived usability, and acceptance of two symptom checkers (Ada and Rheport) in rheumatology: interim results from a randomized controlled crossover trial Arthritis Res Ther 2021 23 1 112 10.1186/s13075-021-02498-8 33849654 PMC8042673 · DOI
- [5]Gräf M Knitza J Leipe J Krusche M Welcker M Kuhn S Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy Rheumatol Int 2022 42 12 2167 2176 10.1007/s00296-022-05202-4 36087130 PMC9548469 · DOI
- [6]Hügle T The wide range of opportunities for large language models such as ChatGPT in rheumatology RMD Open 2023 9 2 e003105 10.1136/rmdopen-2023-003105 37116985 PMC10151992 · DOI
- [7]Thirunavukarasu AJ Ting DSJ Elangovan K Gutierrez L Tan TF Ting DSW Large language models in medicine Nat Med 2023 29 8 1930 1940 10.1038/s41591-023-02448-8 37460753 · DOI · Paper page
- [8]Kung TH Cheatham M Medenilla A Sillos C De Leon L Elepaño C Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models PLOS Digit Health. 2023 2 2 e0000198 10.1371/journal.pdig.0000198 36812645 PMC9931230 · DOI · Paper page
- [9]Thirunavukarasu AJ Hassan R Mahmood S Sanghera R Barzangi K El Mukashfi M Trialling a large language model (ChatGPT) in general practice with the applied knowledge test: observational study demonstrating opportunities and limitations in primary care JMIR Med Educ 2023 9 e46599 10.2196/46599 37083633 PMC10163403 · DOI · Paper page
- [10]Verhoeven F Wendling D Prati C ChatGPT: when artificial intelligence replaces the rheumatologist in medical writing Ann Rheum Dis 2023 82 8 1015 1017 10.1136/ard-2023-223936 37041067 PMC10359572 · DOI · Paper page
- [11]Ueda D Mitsuyama Y Takita H Horiuchi D Walston SL Tatekawa H ChatGPT’s diagnostic performance from patient history and imaging findings on the diagnosis please quizzes Radiology 2023 308 1 e231040 10.1148/radiol.231040 37462501 · DOI
- [12]Kanjee Z Crowe B Rodman A Accuracy of a generative artificial intelligence model in a complex diagnostic challenge JAMA 2023 330 1 78 10.1001/jama.2023.8288 37318797 PMC10273128 · DOI · Paper page
- [14]de Thurah A Bosch P Marques A Meissner Y Mukhtyar CB Knitza J EULAR points to consider for remote care in rheumatic and musculoskeletal diseases Ann Rheum Dis 2022 81 8 1065 1071 10.1136/annrheumdis-2022-222341 35470160 · DOI
- [15]Tools such as ChatGPT threaten transparent science; here are our ground rules for their use. Nature. Januar 2023;613(7945):612. 36694020
Cited by
Sorted by citations
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Takita, H., Kabata, D., Walston, S. L., Tatekawa, H., Saito, K., Tsujimoto, Y., Miki, Y., & Ueda, D. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01543-z · Cited by 203 · see citing works
Ong, J. C. L., Jin, L., Elangovan, K., Lim, G. Y. S., Lim, D. Y. Z., Sng, G. G. R., Ke, Y. H., Tung, J. Y. M., Zhong, R. J., Koh, C. M. Y., et al. (2025) · Cell Reports Medicine · doi:10.1016/j.xcrm.2025.102323 · Cited by 58 · see citing works
Labinsky, H., Klemm, P., Graalmann, L., Thiele, T., Hornig, J., Fink, D., Morf, H., Mucke, J., Kiltz, U., Pecher, A.-C., et al. (2025) · Rheumatology International · doi:10.1007/s00296-025-06023-x · Cited by 20 · see citing works
Self-citationKremer, P., Langballe, E., Haase, I., Bamberger, J., Kuhn, S., Krusche, M., & Knitza, J. (2026) · Rheumatology International · doi:10.1007/s00296-025-06068-y · Cited by 17 · see citing works
Self-citationLechner, F., Kuhn, S., & Knitza, J. (2025) · International Journal of Rheumatic Diseases · doi:10.1111/1756-185x.70124 · Cited by 11 · see citing works
Self-citation
- Authors
- Krusche, M., Callhoff, J., Knitza, J., & Ruffer, N.
- Affiliations
- University Medical Center Hamburg-Eppendorf (DE)
- Philipps University of Marburg (DE)
- Corresponding author
- Krusche, Martin
- Venue
- Rheumatology International
- Published
- 2023-09-24
- Volume
- 44
- Issue
- 2
- Pages
- 303-306
- ISSN
- 0172-8172 (ISSN-L)
- 1437-160X
- PMC
- PMC10796566
- OpenAlex
- W4386998832
- Cited by
- 116
- References
- 1
- Citations by year
- 2026: 36 · 2025: 39 · 2024: 38 · 2023: 3
- FWCI
- 4.62
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Rheumatology International (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Informationszentrum Lebenswissenschaften (ZB MED) (submittedVersion, cc-by)
- Funders
- Universitätsklinikum Hamburg-Eppendorf (UKE)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- ChatGPT-4, large language models, inflammatory rheumatic diseases, diagnostic accuracy, differential diagnosis, rheumatologists, AI-assisted diagnosis, triage, sensitivity and specificity
- MeSH terms
- Rheumatologists, Humans, Rheumatology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
