Comparative analysis of large language models and traditional diagnostic decision support systems for rare rheumatic disease identification
Open accessAbstract
Objectives: Rare rheumatic diseases are challenging to diagnose due to their complex and often atypical presentations. This study compared traditional diagnostic decision support systems (DDSSs) and large language models (LLMs) for rare rheumatic disease identification. Methods: Comparison was based on 60 rare disease vignettes. Four general-purpose LLMs (Claude 3.5 Sonnet, ChatGPT-4o, Gemini 1.5 Pro, and Llama 3.3) and 3 traditional DDSSs (Symptoma, Ada, and Isabel DDx) were used to generate up to 5 disease suggestions per vignette based on anamnestic information. Identical diagnoses were scored with 2 points and plausible diagnoses with 1 point, contributing to a diagnostic score. Case completion time for each vignette was measured, and proportion of identical or plausible diagnoses were calculated. Results: < .001), and the average case diagnostic scores were 3.5 vs 2.1 for LLMs and DDSSs, respectively. LLMs were also more time-efficient than DDSSs with 20 vs 189 seconds per case. Claude 3.5 Sonnet achieved the highest overall diagnostic score (228) based on anamnestic information, followed by ChatGPT-4o (224), Llama 3.3 (200), Gemini 1.5 Pro (187), Symptoma (146), Ada (124), and Isabel DDx (116). Conclusions: General-purpose LLMs outperformed the traditional DDSSs evaluated in this study, including both free and subscription-based medical products, in diagnostic accuracy and efficiency. Notably, open-source, locally hosted LLMs demonstrated promising performance, highlighting the potential of secure, on-premises LLMs for diagnostic decision support.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
General-purpose LLMs outperform traditional diagnostic decision support systems, including long-established certified medical products, in diagnostic accuracy and speed.
“Furthermore, benchmarking work has shown that general-purpose LLMs outperform traditional diagnostic decision support systems (DDSS), including long-established, certified medical products, in terms of diagnostic accuracy but importantly also speed [9].”
From Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation (2026) · cites this paper as [9]
The Discussion reports that the tested LLMs outperformed the tested DDSSs in diagnostic accuracy and time efficiency on rare-disease vignettes. The Traditional DDSSs section says Ada and Symptoma are classified as medical devices, but does not establish that they are certified or long-established. The Discussion also cautions that the focus on rare rheumatic diseases and selected DDSSs limits generalizability, so the claim is too broad without that qualification. (LLM, conf=0.87)
Experimental — accuracy not yet measuredOpenEvidence and Prof. Valmed demonstrated higher proportions of identical top diagnoses than the three traditional diagnostic decision support systems (Isabel DDx, Ada, Symptoma).
“Comparing our findings with the previous comparative study, which evaluated the same set of cases, both OpenEvidence and Prof. Valmed demonstrated higher proportions of identical top diagnoses than the three traditional diagnostic decision support systems (Isabel DDx, Ada, Symptoma).”
From Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation (2026) · cites this paper as [9]
The ‘Overall proportion of identical diagnoses’ section reports that the three traditional systems collectively produced an identical top diagnosis in 8.9% of cases, providing the comparison baseline. The study did not evaluate OpenEvidence or Prof. Valmed, so our copy cannot establish that either had a higher proportion. (LLM, conf=0.86)
Experimental — accuracy not yet measuredAll three LLMs evaluated in this study reported substantially larger probabilities for identical diagnoses than for diagnostically different ones, compared to the four previously analyzed LLMs.
“Importantly, all three LLMs evaluated in this study reported substantially larger probabilities for identical diagnoses than for diagnostically different ones, compared to the four previously analyzed LLMs.”
From Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation (2026) · cites this paper as [9]
The LLMs and Data analysis sections (page unavailable) confirm that the cited paper evaluated four LLMs and calculated top-diagnosis probabilities by diagnostic category. The Diagnostic probability section (page unavailable) does not establish the claimed probability contrast for those four LLMs or a comparison with the citing study’s three LLMs; its stated comparison concerns traditional decision-support systems. (LLM, conf=0.78)
Experimental — accuracy not yet measuredCertified AI-based medical products such as Ada and Symptoma have previously been shown to assign higher probabilities to diagnostically different diagnoses than to identical ones.
“particularly given that certified AI-based medical products such as Ada and Symptoma have previously been shown to assign higher probabilities to diagnostically different diagnoses than to identical ones [9, 27].”
From Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation (2026) · cites this paper as [9]
The Diagnostic probability section reports higher probabilities for diagnostically different suggestions than identical diagnoses for Symptoma (79.3% versus 71.0%). For Ada, it reports the opposite (33.1% versus 45.6%), so the claim is not supported for both products. The Traditional DDSSs section classifies Ada and Symptoma as medical devices but does not establish that they are certified. (LLM, conf=0.94)
Experimental — accuracy not yet measuredThe study adapted previously used methodology to ensure comparability with a recent rheumatology benchmarking study.
“To ensure comparability of the results with a recent rheumatology benchmarking study of the diagnostic accuracy and usability of LLMs and traditional diagnostic decision support systems [9], the previously used methodology was adapted.”
From Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation (2026) · cites this paper as [9]
Results reflect performance under a single clinically motivated prompt and may not generalize across prompting strategies.
“A single standardized prompt was intentionally used to preserve comparability with the benchmark methodology of Kremer et al. [9], however LLM outputs are sensitive to prompt phrasing and structure. Accordingly, results reflect performance under a single clinically motivated prompt and may not generalize across prompting strategies.”
From Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation (2026) · cites this paper as Kremer et al. [9]
ChatGPT-5 Thinking yielded a lower proportion of identical top diagnoses than ChatGPT-4o (26.7% vs. 40.0%), whereas their proportions of identical Top-5 diagnoses were similar (58.3% vs. 60.0%).
“Notably, ChatGPT-5 Thinking yielded a lower proportion of identical top diagnoses than ChatGPT-4o (26.7% vs. 40.0%), whereas their proportions of identical Top-5 diagnoses were similar (58.3% vs. 60.0%).”
From Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation (2026) · cites this paper as [9]
The LLMs section lists the models tested, including ChatGPT-4o, but not ChatGPT-5 Thinking. The Overall proportion of identical diagnoses section reports 40.0% Top-1 and 60.0% Top-5 for Claude 3.5 Sonnet, not as a comparison between the two ChatGPT models. Searches for ChatGPT-5 Thinking and the claimed percentages found no passage establishing that comparison. (LLM, conf=0.78)
Experimental — accuracy not yet measuredOpenEvidence and Prof. Valmed performance remained substantially lower than that of the four large language models analyzed (Gemini, Llama, ChatGPT-4o, and Claude).
“However, their performance remained substantially lower than that of the four large language models analyzed, including Gemini, Llama, ChatGPT-4o, and Claude.”
From Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation (2026) · cites this paper as [9]
The Study design and setting section compares the four named LLMs with Symptoma, Ada, and Isabel DDx—not OpenEvidence or Prof. Valmed. Searches for OpenEvidence and Prof. Valmed found no passage assessing either platform, so their performance relative to the LLMs cannot be verified from this paper. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial
Goh, E., Gallo, R., Hom, J., Strong, E., Weng, Y., Kerman, H., Cool, J. A., Kanjee, Z., Parsons, A. S., Ahuja, N., et al. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.40969 · Cited by 724 · see citing works
Clinical research for rare disease: Opportunities, challenges, and solutions
Griggs, R. C., Batshaw, M., Dunkle, M., Gopal-Srivastava, R., Kaye, E., Krischer, J., Nguyen, T., Paulus, K., & Merkel, P. A. (2009) · Molecular Genetics and Metabolism · doi:10.1016/j.ymgme.2008.10.003 · Cited by 401 · see citing works
The Global Burden of Diagnostic Errors in Primary Care
Singh, H., Schiff, G. D., Graber, M. L., Onakpoya, I., & Thompson, M. J. (2017) · BMJ Quality & Safety · doi:10.1136/bmjqs-2016-005401 · Cited by 395 · see citing works
Flagged first, then by citations
Research works printed but not linked: 34
- Richens JG, Lee CM, Johri S. Improving the accuracy of medical diagnosis with causal machine learning. Nat Commun 2020;11:3923.
- Committee on Diagnostic Error in Health Care, Board on Health Care Serv- ices, Institute of Medicine, The National Academies of Sciences, Engineering, and Medicine. In: Balogh EP, Miller BT, Ball JR, editors. Improving Diagnosis in Health Care [Internet]. Washington, DC: National Academies Press; 2015. p. 472 p. [cited 2024 Nov 27]. Available from:. doi: 10.17226/21794. · DOI · Paper page
- Willmen L, V€ olkel L, Willmen T, Deckersbach T, Geyer S, Wagner AD. The economic burden of diagnostic uncertainty on rare disease patients. BMC Health Serv Res 2024;24:1388.
- European Commission. Directorate General for Health and Food Safety. Euro- pean Reference Networks: Working for patients with rare, low prevalence and complex diseases : share, care, cure. [Internet]. Luxembourg: Publica- tions Office of the European Union; 2023. p. 44 p. [cited 2024 Nov 27]Avail- able from: https://data.europa.eu/doi/10.2875/701174. · DOI
- Halley MC, Smith HS, Ashley EA, Goldenberg AJ, Tabor HK. A call for an integrated approach to improve efficiency, equity and sustainability in rare disease research in the United States. Nat Genet 2022;54:219-22.
- Do Nascimento RRNR, Piotto DGP, Freire EAM, De Souza Neves F, Sztajnbok FR, Bica BERG, et al. Rare diseases: what rheumatologists need to know? Adv Rheumatol 2024;64:74.
- Goldblatt F, O, Neill SG. Clinical aspects of autoimmune rheumatic diseases. Lancet 2013;382:797-808.
- Stoller JK. The challenge of rare diseases. Chest 2018;153:1309-14.
- Faviez C, Chen X, Garcelon N, Neuraz A, Knebelmann B, Salomon R, et al. Diagnosis support systems for rare diseases: a scoping review. Orphanet J Rare Dis 2020;15:94.
- Gr€ af M, Knitza J, Leipe J, Krusche M, Welcker M, Kuhn S, et al. Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy. Rheumatol Int 2022;42:2167-76.
- Moens HJ, van der Korst JK. Computer-assisted diagnosis of rheumatic disor- ders. Semin Arthritis Rheum 1991;21:156-69.
- M€ uhlensiepen F, Kurkowski S, Krusche M, Mucke J, Prill R, Heinze M, et al. Digital health transition in rheumatology: a qualitative study. Int J Environ Res Public Health 2021;18:2636.
- Knitza J, Muehlensiepen F, Ignatyev Y, Fuchs F, Mohn J, Simon D, et al. Patient's perception of digital symptom assessment technologies in rheumatol- ogy: results from a multicentre study. Front Public Health 2022;10:844669.
- Kernder A, Morf H, Klemm P, Vossen D, Haase I, Mucke J, et al. Digital rheu- matology in the era of COVID-19: results of a national patient and physician survey. RMD Open 2021;7:e001548.
- Alder H, Michel BA, Marx C, Tamborrini G, Langenegger T, Bruehlmann P, et al. Computer-based diagnostic expert systems in rheumatology: where do we stand in 2014? Int J Rheumatol 2014;2014:672714.
- Knitza J, Tascilar K, Gruber E, Kaletta H, Hagen M, Liphardt AM, et al. Accu- racy and usability of a diagnostic decision support system in the diagnosis of three representative rheumatic diseases: a randomized controlled trial among medical students. Arthritis Res Ther 2021;23:233.
- Schwartz WB. Medicine and the computer. The promise and problems of change. N Engl J Med 1970;283:1257-64.
- Munsch N, Martin A, Gruarin S, Nateqi J, Abdarahmane I, Weingartner-Ort- ner R, et al. Diagnostic accuracy of web-based COVID-19 symptom checkers: comparison study. J Med Internet Res 2020;22:e21299.
- Abdullahi T, Singh R, Eickhoff C. Learning to make rare and complex diagno- ses with generative AI assistance: qualitative study of popular large language models. JMIR Med Educ 2024;10:e51391.
- Chen X, Mao X, Guo Q, Wang L, Zhang S, Chen T. RareBench: can LLMs serve as rare diseases specialists? In: Baeza-Yates R, Bonchi F, editors. In: Proceed- ings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; 2024 Aug 25-29; 2024. p. Barcelona, Spain: ACM Digital Library 4850-61. https://dl.acm.org/doi/10.1145/3637528.3671576. · DOI · Paper page
- Knitza J, Tascilar K, Fuchs F, Mohn J, Kuhn S, Bohr D, et al. Diagnostic accu- racy of a mobile AI-based symptom checker and a web-based self-referral tool in rheumatology: multicenter randomized controlled trial. J Med Inter- net Res 2024;26:e55542.
- Nateqi J, Lin S, Krobath H, Gruarin S, Lutz T, Dvorak T, et al. Vom Symptom zur Diagnose -Tauglichkeit von Symptom-Checkern : Update aus Sicht der HNO [From symptom to diagnosis-symptom checkers re-evaluated: are symp- tom checkers finally sufficient and accurate to use? An update from the ENT perspective]. HNO. 2019;67:334-42.
- Riches N, Panagioti M, Alam R, Cheraghi-Sohi S, Campbell S, Esmail A, et al. The effectiveness of electronic differential diagnoses (DDX) generators: a sys- tematic review and meta-analysis. PLoS One 2016;11:e0148991.
- Hautz WE, K€ ammer JE, Hautz SC, Sauter TC, Zwaan L, Exadaktylos AK, et al. Diagnostic error increases mortality and length of hospital stay in patients presenting through the emergency room. Scand J Trauma Resusc Emerg Med 2019;27:54.
- Bastakoti M, Muhailan M, Nassar A, Sallam T, Desale S, Fouda R, et al. Dis- crepancy between emergency department admission diagnosis and hospital discharge diagnosis and its impact on length of stay, up-triage to the inten- sive care unit, and mortality. Diagnosis (Berl) 2021;9:107-14.
- Ehrenstein B, Pongratz G, Fleck M, Hartung W. The ability of rheumatologists blinded to prior workup to diagnose rheumatoid arthritis only by clinical assessment: a cross-sectional study. Rheumatology 2018;57:1592-601.
- Bridges JM. Computerized diagnostic decision support systems -a com- parative performance study of Isabel Pro vs. ChatGPT4. Diagnosis 2024;11:250-8.
- Hirosawa T, Harada Y, Tokumasu K, Ito T, Suzuki T, Shimizu T. Comparative study to evaluate the accuracy of differential diagnosis lists generated by Gemini Advanced, Gemini, and Bard for a case report series analysis: cross- sectional study. JMIR Med Inform 2024;12:e63010.
- Schlicker N, Langer M, Hirsch MC. Wie vertrauensw€ urdig ist k€ unstliche Intel- ligenz? : Ein Modell f€ ur das Spannungsfeld zwischen Objektivit€ at und Sub- jektivit€ at [How trustworthy is artificial intelligence?: a model for the conflict between objectivity and subjectivity]. Inn Med (Heidelb) 2023;64:1051-7.
- Freyer O, Wiest IC, Kather JN, Gilbert S. A future role for health applications of large language models depends on regulators enforcing safety standards. Lancet Digit Health 2024;6:e662-72.
- Reese JT, Chimirri L, Bridges Y, Danis D, Caufield JH, Wissink K, et al. System- atic benchmarking demonstrates large language models have not reached the diagnostic accuracy of traditional rare-disease decision support tools. medRxiv 2024.07.22.24310816 [Preprint]. 2024 [cited 2025 Apr 14]: [16 p.]. Available from: https://www.medrxiv.org/content/10.1101/2024.07.22.24310816v2 · DOI
- Yang J, Shu L, Duan H, Li H. A robust phenotype-driven likelihood ratio anal- ysis approach assisting interpretable clinical diagnosis of rare diseases. J Biomed Inform 2023;142:104372.
- Yang J, Shu L, Han M, Pan J, Chen L, Yuan T, et al. RDmaster: a novel pheno- type-oriented dialogue system supporting differential diagnosis of rare dis- ease. Comput Biol Med 2024;169:107924.
- Forrest IS, Petrazzini BO, A Duffy, Park JK, O'Neal AJ, Jordan DM, et al. A machine learning model identifies patients in need of autoimmune disease testing using electronic health records. Nat Commun 2023;14:2385.
Cited by
Sorted by citations
Labinsky, H., Klemm, P., Graalmann, L., Thiele, T., Hornig, J., Fink, D., Morf, H., Mucke, J., Kiltz, U., Pecher, A.-C., et al. (2025) · Rheumatology International · doi:10.1007/s00296-025-06023-x · Cited by 20 · see citing works
Self-citationKremer, P., Langballe, E., Haase, I., Bamberger, J., Kuhn, S., Krusche, M., & Knitza, J. (2026) · Rheumatology International · doi:10.1007/s00296-025-06068-y · Cited by 17 · see citing works
“To ensure comparability of the results with a recent rheumatology benchmarking study of the diagnostic accuracy and usability of LLMs and traditional diagnostic decision support systems [9], the previously used methodology was adapted.”
InfluentialSelf-citation
- Authors
- Kremer, P., Schiebisch, H., Lechner, F., Haase, I., Cirkel, L., Kötter, I., Kuhn, S., Krusche, M., & Knitza, J.
- Affiliations
- University Medical Center Hamburg-Eppendorf (DE)
- Philipps University of Marburg (DE)
- Corresponding author
- Knitza, Johannes
- Venue
- EULAR Rheumatology Open
- Publisher
- Elsevier BV
- Published
- 2025-06
- Volume
- 1
- Issue
- 2
- Pages
- 51-59
- ISSN
- 3050-7081 (ISSN-L)
- PMC
- PMC13292275
- OpenAlex
- W4410460999
- Cited by
- 17
- References
- 3
- Citations by year
- 2026: 13 · 2025: 4
- FWCI
- 4.09
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- EULAR Rheumatology Open (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion)
- Funders
- European Commission (101080711)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- large language models, diagnostic decision support systems, disease identification, clinical vignettes, diagnostic accuracy, diagnostic efficiency, Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro, Llama 3.3, Ada
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
