CoChat
Sign Up
Cited by 17Artificial Intelligence in Healthcare and EducationArtificial Intelligence in HealthcareClinical Reasoning and Diagnostic Skills

Diagnostic performance of Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence in rheumatology: A comparative evaluation

Open access
Kremer, Phillip, Langballe, Emily, Haase, Isabell, Bamberger, Jonathan, Kuhn, Sebastian, Krusche, Martin, Knitza, Johannes·2026·Rheumatology International
DOI 10.1007/s00296-025-06068-yPMC PMC12790495
Open this paper in CoChat — free accountOpen PDF

Abstract

To compare the diagnostic performance of a subscription-based medical large language model (LLM) certified as a medical device (Prof. Valmed), a subscription-based general-purpose LLM (ChatGPT-5 Thinking), and a freely accessible medical LLM (OpenEvidence). Sixty vignettes covering rare rheumatic diseases and differential diagnoses were entered using a standardized prompt to generate five top diagnoses and respective diagnostic probabilities. Blinded rheumatologists categorized suggested diagnoses as identical, plausible, or diagnostically different. Diagnostic accuracy was assessed using proportions of identical and plausible diagnoses and a total diagnostic score. Analyses were descriptive. Group differences for the proportion of identical top diagnoses were explored using Cochran’s Q test and post-hoc McNemar tests. Processing time was recorded. OpenEvidence produced the highest proportion of identical top diagnoses (35.0%), followed by ChatGPT-5 Thinking (26.7%) and Prof. Valmed (23.3%); however, pairwise differences were not statistically significant. ChatGPT-5 Thinking achieved the highest total diagnostic score (226), followed by OpenEvidence (221) and Prof. Valmed (212). All systems showed markedly higher diagnostic probabilities for identical versus different diagnoses. Mean processing times ranged from 20 to 36 s. All three systems demonstrated broadly comparable diagnostic accuracy and processing times. Further benchmarking that incorporates additional evaluation dimensions is essential to inform safe and effective use of LLM-based clinical support.

AI snapshot

Written from the paper’s full text.

TL;DR

This comparative benchmarking study found broadly comparable diagnostic accuracy and processing times among Prof. Valmed, ChatGPT-5 Thinking, and OpenEvidence on 60 rheumatology vignettes. OpenEvidence had the highest proportion of identical top diagnoses, but pairwise differences were not statistically significant.

Key findings

  • OpenEvidence produced the highest proportion of identical top diagnoses (35.0%; Top5 56.7%), followed by ChatGPT-5 Thinking (26.7%; Top5 58.3%) and Prof. Valmed (23.3%; Top5 51.7%).
  • ChatGPT-5 Thinking achieved the highest total diagnostic score (226), followed by OpenEvidence (221) and Prof. Valmed (212).
  • Mean processing times were 20 s for Prof. Valmed, 36 s for ChatGPT-5 Thinking, and 31 s for OpenEvidence.
  • Cochran’s Q indicated a significant overall difference among systems (p = 0.001), but post-hoc McNemar tests with Holm correction found no significant pairwise differences (Prof. Valmed vs OpenEvidence p = 0.0654; ChatGPT-5 Thinking vs OpenEvidence p = 0.1797; Prof. Valmed vs ChatGPT-5 Thinking p = 0.7905). All systems reported higher diagnostic probabilities for identical than for diagnostically different diagnoses.

Methods

  • Sixty vignettes covering rare rheumatic diseases and differential diagnoses were used: 50 sourced from high-impact journals and 10 based on real-world cases from the Department of Rheumatology at University Hospital Hamburg-Eppendorf, Germany.
  • A standardized prompt was sequentially entered into each system’s web interface between August 25 and August 27, 2025, with a new session per vignette; systems generated five top diagnoses with diagnostic probabilities, and processing time was recorded.
  • Three blinded rheumatologists independently categorized diagnoses as identical, plausible, or diagnostically different (Cohen’s κ = 0.907 for top diagnoses and 0.846 for all diagnoses), with a fourth board-certified rheumatologist adjudicating disagreements. Analyses were descriptive; Cochran’s Q and post-hoc McNemar tests with Holm correction were used for proportions of identical top diagnoses.

Limitations

  • The paper states possible “data leakage,” since LLMs may have been exposed to the previously published benchmarking vignette collection or original case reports; no model cited the earlier publication or original case reports, but absence of citations does not rule out exposure.
  • The paper states that a single standardized prompt was used for comparability, so results may not generalize across prompting strategies; no a priori power calculation was done, so the study may be underpowered for small between-model differences.
  • The paper states that internal consistency was not assessed, that vignette-based studies require real-world testing with end users, and that the study is a single-time snapshot given rapid LLM evolution. Evident from the text: the evaluation is limited to text input and does not assess image-based analysis, and usability/reference quality differences were noted but not formally scored.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s00296-025-06068-y

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

24 of 30 cited sources checked · 9 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    General-purpose LLMs such as ChatGPT can achieve diagnostic accuracy comparable to that of experienced rheumatologists.

    “Krusche et al. demonstrated in 2023 that general-purpose LLMs, such as ChatGPT, can achieve diagnostic accuracy comparable to that of experienced rheumatologists [10].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [10] Krusche M, Callhoff J, Knitza J et al (2023) Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4. Rheumatol Int. https://doi.org/10.1007/s00296-023-05464-6 source
  • Premisecitation

    General-purpose LLMs outperform traditional diagnostic decision support systems, including long-established certified medical products, in diagnostic accuracy and speed.

    “Furthermore, benchmarking work has shown that general-purpose LLMs outperform traditional diagnostic decision support systems (DDSS), including long-established, certified medical products, in terms of diagnostic accuracy but importantly also speed [9].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [9] Kremer P, Schiebisch H, Lechner F et al (2025) Comparative analysis of large language models and traditional diagnostic decision support systems for rare rheumatic disease identification. EULAR Rheumatol Open 1:51–59. https://doi.org/10.1016/j.ero.2025.04.007 source

      The Discussion reports that the tested LLMs outperformed the tested DDSSs in diagnostic accuracy and time efficiency on rare-disease vignettes. The Traditional DDSSs section says Ada and Symptoma are classified as medical devices, but does not establish that they are certified or long-established. The Discussion also cautions that the focus on rare rheumatic diseases and selected DDSSs limits generalizability, so the claim is too broad without that qualification. (LLM, conf=0.87)

      Experimental — accuracy not yet measured
  • Findingcitation

    OpenEvidence and Prof. Valmed demonstrated higher proportions of identical top diagnoses than the three traditional diagnostic decision support systems (Isabel DDx, Ada, Symptoma).

    “Comparing our findings with the previous comparative study, which evaluated the same set of cases, both OpenEvidence and Prof. Valmed demonstrated higher proportions of identical top diagnoses than the three traditional diagnostic decision support systems (Isabel DDx, Ada, Symptoma).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [9] Kremer P, Schiebisch H, Lechner F et al (2025) Comparative analysis of large language models and traditional diagnostic decision support systems for rare rheumatic disease identification. EULAR Rheumatol Open 1:51–59. https://doi.org/10.1016/j.ero.2025.04.007 source

      The ‘Overall proportion of identical diagnoses’ section reports that the three traditional systems collectively produced an identical top diagnosis in 8.9% of cases, providing the comparison baseline. The study did not evaluate OpenEvidence or Prof. Valmed, so our copy cannot establish that either had a higher proportion. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Findingown data

    All three LLMs evaluated in this study reported substantially larger probabilities for identical diagnoses than for diagnostically different ones, compared to the four previously analyzed LLMs.

    “Importantly, all three LLMs evaluated in this study reported substantially larger probabilities for identical diagnoses than for diagnostically different ones, compared to the four previously analyzed LLMs.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [9] Kremer P, Schiebisch H, Lechner F et al (2025) Comparative analysis of large language models and traditional diagnostic decision support systems for rare rheumatic disease identification. EULAR Rheumatol Open 1:51–59. https://doi.org/10.1016/j.ero.2025.04.007 source

      The LLMs and Data analysis sections (page unavailable) confirm that the cited paper evaluated four LLMs and calculated top-diagnosis probabilities by diagnostic category. The Diagnostic probability section (page unavailable) does not establish the claimed probability contrast for those four LLMs or a comparison with the citing study’s three LLMs; its stated comparison concerns traditional decision-support systems. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
  • Findingown data

    OpenEvidence and Prof. Valmed performance remained substantially lower than that of the four large language models analyzed (Gemini, Llama, ChatGPT-4o, and Claude).

    “However, their performance remained substantially lower than that of the four large language models analyzed, including Gemini, Llama, ChatGPT-4o, and Claude.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [9] Kremer P, Schiebisch H, Lechner F et al (2025) Comparative analysis of large language models and traditional diagnostic decision support systems for rare rheumatic disease identification. EULAR Rheumatol Open 1:51–59. https://doi.org/10.1016/j.ero.2025.04.007 source

      The Study design and setting section compares the four named LLMs with Symptoma, Ada, and Isabel DDx—not OpenEvidence or Prof. Valmed. Searches for OpenEvidence and Prof. Valmed found no passage assessing either platform, so their performance relative to the LLMs cannot be verified from this paper. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 25
  • May combine two claims. 11

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    A total diagnostic score adapted from Goh et al. was applied, assigning two points for identical and one point for plausible diagnoses.

    “To further quantify diagnostic performance, a total diagnostic score, adapted from Goh et al. [18], was applied, assigning two points for identical and one point for plausible diagnoses.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Goh et al. [18] Goh E, Gallo R, Hom J et al (2024) Large language model influence on diagnostic reasoning: a randomized clinical trial. JAMA Netw Open 7:e2440969. https://doi.org/10.1001/jamanetworkopen.2024.40969 source

      In Methods > Assessing Performance, Goh et al. award 2 points for the most correct final diagnosis and 1 point for a plausible or correct but insufficiently specific diagnosis. That supports adapting a two-/one-point diagnosis score, but the paper does not use “identical” as its two-point criterion; its total score also includes other components of the structured reflection grid (Methods > Study Outcomes). (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Rheumatic patients and rheumatologists both agree that diagnostic decision support is the most relevant field of implementation in rheumatology for LLMs.

    “surveys demonstrated that rheumatic patients and rheumatologists both agree that diagnostic decision support is the most relevant field of implementation in rheumatology for LLMs [6, 12].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [6] Holzer M-T, Meinecke A, Müller F et al (2024) Artificial intelligence in rheumatology: status quo and quo vadis-results of a national survey among German rheumatologists. Ther Adv Musculoskelet Dis 16:1759720X241275818. https://doi.org/10.1177/1759720X241275818 source

      The Results (AI usage; page not provided) show that diagnosis was rheumatologists’ most frequently selected future application for AI (72.7%). But this survey did not include patients, and its LLM-specific results placed diagnosis behind scientific writing and patient correspondence. It therefore does not establish that both groups ranked diagnostic decision support as the most relevant use for LLMs. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites [12] Labinsky H, Klemm P, Graalmann L et al (2025) Patient experiences, attitudes, and profiles regarding artificial intelligence in rheumatology: a German national cross-sectional survey study. Rheumatol Int 45:269. https://doi.org/10.1007/s00296-025-06023-x source

      The Results (Current and Potential Use Cases of AI in Rheumatology; page unavailable) report that patients were most interested in AI-based symptom checkers, and the Conclusion says patients and rheumatologists share priorities around diagnostics and education. The paper does not establish that both groups ranked diagnostic decision support as the single most relevant application, or that their rankings concerned LLMs specifically rather than AI more broadly. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Medical domain-specific tools increasingly use LLMs to retrieve verified medical information and, through retrieval-augmented generation, translate this validated content into personalized answers rather than relying solely on training data.

    “Medical domain-specific tools increasingly use LLMs to retrieve verified medical information and, through retrieval-augmented generation (RAG), translate this validated content into personalized answers, rather than relying solely on training data [8].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [8] Lechner F, Kuhn S, Knitza J (2025) Harnessing large language models for rheumatic disease diagnosis: advancing hybrid care and task shifting. Int J Rheum Dis 28:e70124. https://doi.org/10.1111/1756-185X.70124 source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In a small retrospective evaluation, physicians rated OpenEvidence's answers highly in terms of clarity, clinical relevance, and evidence-based support.

    “In a small retrospective evaluation, physicians rated its answers highly in terms of clarity, clinical relevance, and evidence-based support [14].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [14] Hurt RT, Stephenson CR, Gilman EA et al (2025) The use of an artificial intelligence platform OpenEvidence to augment clinical decision-making for primary care physicians. J Prim Care Commun Health 16:21501319251332215. https://doi.org/10.1177/21501319251332215 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The swift integration of LLMs into clinical workflows has led to the release of first official guidelines for their use in practice.

    “The swift integration of LLMs into clinical workflows has also led to the release of first official guidelines for their use in practice [11].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [11] Wong EYT, Verlingue L, Aldea M et al (2025) ESMO guidance on the use of Large Language Models in Clinical Practice (ELCAP). Ann Oncol. https://doi.org/10.1016/j.annonc.2025.09.001 source

      The Introduction (page unavailable) describes rapid LLM uptake and says ESMO developed guidance for their use in oncology; the practical-guidance section provides recommendations for clinicians. The paper does not establish that these were the first official guidelines, and its guidance is specific to oncology rather than clinical practice generally. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In a recent large national survey, the majority of patients stated that they would welcome their rheumatologists using AI for clinical decision support.

    “Importantly, in a recent large national survey, the majority of patients stated that they would welcome their rheumatologists using AI for clinical decision support [12].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [12] Labinsky H, Klemm P, Graalmann L et al (2025) Patient experiences, attitudes, and profiles regarding artificial intelligence in rheumatology: a German national cross-sectional survey study. Rheumatol Int 45:269. https://doi.org/10.1007/s00296-025-06023-x source

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    The study adapted previously used methodology to ensure comparability with a recent rheumatology benchmarking study.

    “To ensure comparability of the results with a recent rheumatology benchmarking study of the diagnostic accuracy and usability of LLMs and traditional diagnostic decision support systems [9], the previously used methodology was adapted.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [9] Kremer P, Schiebisch H, Lechner F et al (2025) Comparative analysis of large language models and traditional diagnostic decision support systems for rare rheumatic disease identification. EULAR Rheumatol Open 1:51–59. https://doi.org/10.1016/j.ero.2025.04.007 source

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Diagnostic suggestions were classified according to procedures described by Hautz et al. and Bastakoti et al.

    “Diagnostic suggestions were classified by the raters as identical, plausible, or diagnostically different, according to the procedures described by Hautz et al. [16] and Bastakoti et al. [17].”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Hautz et al. [16] Hautz WE, Kämmer JE, Hautz SC et al (2019) Diagnostic error increases mortality and length of hospital stay in patients presenting through the emergency room. Scand J Trauma Resusc Emerg Med 27:54. https://doi.org/10.1186/s13049-019-0629-z source

      In Methods > Measures (page not available), Hautz et al. describe expert raters using a predefined scheme to classify the relationship between admission and discharge diagnoses. Table 1 includes an ‘Identical’ category, but the paper does not describe classifying diagnostic suggestions into the three stated categories—identical, plausible, and diagnostically different. The Bastakoti citation is outside the scope of this verdict. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Bastakoti et al. [17] Bastakoti M, Muhailan M, Nassar A et al (2021) Discrepancy between emergency department admission diagnosis and hospital discharge diagnosis and its impact on length of stay, up-triage to the intensive care unit, and mortality. Diagnosis 9:107–114. https://doi.org/10.1515/dx-2021-0001 source

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Barriers to the clinical implementation of LLMs include their susceptibility to hallucinations and the absence of reliable references.

    “Barriers to the clinical implementation of LLMs include their susceptibility to hallucinations, the absence of reliable references [13].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [13] Moor M, Banerjee O, Abad ZSH et al (2023) Foundation models for generalist medical artificial intelligence. Nature 616:259–265. https://doi.org/10.1038/s41586-023-05881-4 source

      The “Challenges of GMAI” section warns that models may confidently fabricate inaccurate information and says fact-checking their outputs is a serious challenge. It does not say that reliable references are absent; instead, it proposes links to supporting literature as a way to make verification easier. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingcitation

    ChatGPT-5 Thinking yielded a lower proportion of identical top diagnoses than ChatGPT-4o (26.7% vs. 40.0%), whereas their proportions of identical Top-5 diagnoses were similar (58.3% vs. 60.0%).

    “Notably, ChatGPT-5 Thinking yielded a lower proportion of identical top diagnoses than ChatGPT-4o (26.7% vs. 40.0%), whereas their proportions of identical Top-5 diagnoses were similar (58.3% vs. 60.0%).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [9] Kremer P, Schiebisch H, Lechner F et al (2025) Comparative analysis of large language models and traditional diagnostic decision support systems for rare rheumatic disease identification. EULAR Rheumatol Open 1:51–59. https://doi.org/10.1016/j.ero.2025.04.007 source

      The LLMs section lists the models tested, including ChatGPT-4o, but not ChatGPT-5 Thinking. The Overall proportion of identical diagnoses section reports 40.0% Top-1 and 60.0% Top-5 for Claude 3.5 Sonnet, not as a comparison between the two ChatGPT models. Searches for ChatGPT-5 Thinking and the claimed percentages found no passage establishing that comparison. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Study design and conductreasoning

    Results reflect performance under a single clinically motivated prompt and may not generalize across prompting strategies.

    “A single standardized prompt was intentionally used to preserve comparability with the benchmark methodology of Kremer et al. [9], however LLM outputs are sensitive to prompt phrasing and structure. Accordingly, results reflect performance under a single clinically motivated prompt and may not generalize across prompting strategies.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Kremer et al. [9] Kremer P, Schiebisch H, Lechner F et al (2025) Comparative analysis of large language models and traditional diagnostic decision support systems for rare rheumatic disease identification. EULAR Rheumatol Open 1:51–59. https://doi.org/10.1016/j.ero.2025.04.007 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A study by Low et al. using 50 clinical questions showed that special-purpose LLMs, including OpenEvidence, consistently outperformed general-purpose models in generating reliable answers.

    “Similarly, a study by Low et al. using 50 clinical questions showed that special-purpose LLMs, including OpenEvidence, consistently outperformed general-purpose models in generating reliable answers [29].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [29] Low YS, Jackson ML, Hyde RJ et al (2025) Answering real-world clinical questions using large language model, retrieval-augmented generation, and agentic systems. Digit Health 11:20552076251348850. https://doi.org/10.1177/20552076251348850 source

      The Background and significance section says the study tested 50 clinical questions, and the Results section reports higher rates of relevant, evidence-based answers for OpenEvidence and ChatRWD than for the general-purpose models. However, “consistently outperformed” overstates the pairwise results: OpenEvidence significantly outperformed Gemini, but not the other general-purpose models. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A recent comparative study of 15 LLMs on osteoarticular infections found that OpenEvidence achieved the highest overall score.

    “These findings further align with a recent comparative study of 15 LLMs on osteoarticular infections, in which OpenEvidence achieved the highest overall score [30].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [30] Borgonovo F, Matsuo T, Petri F et al (2025) Battle of the bots: solving clinical cases in osteoarticular infections with large language models. Mayo Clin Proc Digit Health 3:100230. https://doi.org/10.1016/j.mcpdig.2025.100230 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    OpenEvidence provided significantly more accurate answers on skin malignancies compared to ChatGPT.

    “Another evaluation found that OpenEvidence provided significantly more accurate answers on skin malignancies compared to ChatGPT [28].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [28] Nihal A, Yang K, Pavlidakey PG. OpenEvidence Is Superior to GPT in Skin Cancer Guideline Queries. Int J Dermatol. Published Online First: 5 June 2025. https://doi.org/10.1111/ijd.17893 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Certified AI-based medical products such as Ada and Symptoma have previously been shown to assign higher probabilities to diagnostically different diagnoses than to identical ones.

    “particularly given that certified AI-based medical products such as Ada and Symptoma have previously been shown to assign higher probabilities to diagnostically different diagnoses than to identical ones [9, 27].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites [9] Kremer P, Schiebisch H, Lechner F et al (2025) Comparative analysis of large language models and traditional diagnostic decision support systems for rare rheumatic disease identification. EULAR Rheumatol Open 1:51–59. https://doi.org/10.1016/j.ero.2025.04.007 source

      The Diagnostic probability section reports higher probabilities for diagnostically different suggestions than identical diagnoses for Symptoma (79.3% versus 71.0%). For Ada, it reports the opposite (33.1% versus 45.6%), so the claim is not supported for both products. The Traditional DDSSs section classifies Ada and Symptoma as medical devices but does not establish that they are certified. (LLM, conf=0.94)

      Experimental — accuracy not yet measured
    • cites [27] Knitza J, Tascilar K, Gruber E et al (2021) Accuracy and usability of a diagnostic decision support system in the diagnosis of three representative rheumatic diseases: a randomized controlled trial among medical students. Arthritis Res Ther 23:233. https://doi.org/10.1186/s13075-021-02616-6 source

      The Results section (page unavailable) reports that Ada assigned higher mean probabilities to incorrect than to correct diagnoses in a substantial number of situations. The Introduction identifies Ada as CE-certified, but this study does not evaluate Symptoma; its findings also concern three rheumatic-disease case vignettes, not diagnostic products generally. (LLM, conf=0.87)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    This pattern aligns with previous observations that the performance of ChatGPT models may decline over time.

    “This pattern aligns with previous observations that the performance of ChatGPT models may decline over time [26].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [26] Griewing S, Knitza J, Boekhoff J et al (2024) Evolution of publicly available large language models for complex decision-making in breast cancer care. Arch Gynecol Obstet 310:537–550. https://doi.org/10.1007/s00404-024-07565-4 source

  • May combine two claims.

    Findingown data

    ChatGPT achieved the highest total diagnostic score (226), followed by OpenEvidence (221) and Prof. Valmed (212).

    “ChatGPT achieved the highest total diagnostic score (226), followed by OpenEvidence (221) and Prof. Valmed (212)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Mean processing times were 20 s for Prof. Valmed, 36 s for ChatGPT, and 31 s for OpenEvidence.

    “Mean processing times were 20 s for Prof. Valmed, 36 s ChatGPT, and 31 s for OpenEvidence”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    OpenEvidence achieved the highest proportion of identical top diagnoses (Top1: 35.0%, Top5: 56.7%), followed by ChatGPT (Top1: 26.7%, Top5: 58.3%), and Prof. Valmed (Top1: 23.3%, Top5: 51.7%).

    “OpenEvidence achieved the highest proportion of identical top diagnoses (Top1: 35.0%, Top5: 56.7%), followed by ChatGPT (Top1: 26.7%, Top5: 58.3%), and Prof. Valmed, (Top1: 23.3%, Top5: 51.7%)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Processing times for all systems averaged under one minute.

    “Processing times for all systems averaged under one minute”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    ChatGPT-5 Thinking did not provide references.

    “Overall usability differences between the systems included that ChatGPT-5 Thinking did not provide references.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    The interrater agreement was very strong, with Cohen's κ of 0.907 for top diagnoses and 0.846 for all diagnoses.

    “The interrater agreement was very strong, with Cohen’s κ of 0.907 for top diagnoses and 0.846 for all diagnoses.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    A new session was created for each vignette to ensure that the models did not retain information from previous interactions that could influence their responses.

    “A new session was created for each vignette to ensure that the models did not retain information from previous interactions that could influence their responses.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The suggested diagnoses were independently and blindly reviewed by three rheumatologists from three different university hospital centers.

    “The suggested diagnoses were independently and blindly reviewed by three rheumatologists from three different university hospital centers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The three systems demonstrated broadly comparable performance with respect to diagnostic accuracy, diagnostic probabilities, and processing time.

    “All three systems demonstrated broadly comparable diagnostic accuracy and processing times.”
    • Also stated in: § Conclusion · boosted
      “demonstrated broadly comparable performance with respect to diagnostic accuracy, diagnostic probabilities, and processing time.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The same prompt containing the vignette information was sequentially input using the respective web interfaces between August 25 and August 27, 2025.

    “The same prompt (see supplementary file 1) containing the vignette information was sequentially input using the respective web interfaces between August 25 and August 27, 2025.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    In cases of disagreement, a fourth board-certified rheumatologist adjudicated the final categorization.

    “In cases of disagreement, a fourth board-certified rheumatologist adjudicated the final categorization.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Even when suitable documents are retrieved, the model may not reliably weave them into its answer, which can blunt any observable improvement in diagnostic performance.

    “And even when suitable documents are retrieved, the model may not reliably weave them into its answer, which can blunt any observable improvement in diagnostic performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    This study represents only a snapshot of system performance at a single point in time.

    “Finally, because this study represents only a snapshot of system performance at a single point in time, and given the rapid evolution of LLMs, periodic re-evaluation will be necessary to maintain an up-to-date understanding of their capabilities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Internal consistency was not assessed in the present study.

    “Internal consistency, however, was not assessed in the present study.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    Data leakage (LLMs exposed to previously published benchmarking vignette collection or original case reports) is possible and difficult to exclude entirely.

    “A potential limitation of this study is the possibility that LLMs may have been exposed to the previously published benchmarking vignette collection or the original published case reports. Such “data leakage” is an inherent benchmarking challenge and is difficult to exclude entirely.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Medical-device certification is designed to secure safe and reliable use through risk management, traceability, and rigorous quality assurance across the product lifecycle.

    “Medical-device certification is therefore designed to secure safe and reliable use through risk management, traceability, and rigorous quality assurance across the product lifecycle [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] Freyer O, Jayabalan S, Kather JN et al (2025) Overcoming regulatory barriers to the implementation of AI agents in healthcare. Nat Med 31:3239–3243. https://doi.org/10.1038/s41591-025-03841-1 source

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The absence of explicit citations does not rule out such exposure, particularly for general-purpose LLMs.

    “However, the absence of explicit citations does not rule out such exposure, particularly for general-purpose LLMs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    No a priori power calculation was done, so the study may be underpowered for small between-model differences.

    “Given the exploratory aim and lack of prior effect-size estimates, no a priori power calculation was done, so the study may be underpowered for small between-model differences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The delay from processing times is clinically negligible.

    “making the delay clinically negligible.”
    Quote in the paperNot a substantial claim
35 of 35

  • The paper relies on this without support.

    The 60 vignettes are representative of rare rheumatic diseases and differential diagnoses encountered in rheumatology.

    “60 representative vignettes encompassing rare rheumatic diseases and differential diagnoses were used.”

  • The paper relies on this without support.

    The standardized prompt used adequately reflects real-world clinical queries.

    “A single standardized prompt was intentionally used to preserve comparability with the benchmark methodology of Kremer et al. [9], however LLM outputs are sensitive to prompt phrasing and structure.”

  • The paper relies on this without support.

    Comparative evidence on whether design and regulatory features translate into measurable diagnostic benefit is limited.

    “However, comparative evidence on whether such design and regulatory features translate into measurable diagnostic benefit is limited.”

  • The paper relies on this without support.

    Head-to-head evaluations that include certified, RAG-enhanced medical products alongside general-purpose LLMs are lacking in rheumatology.

    “In particular, head-to-head evaluations that include certified, RAG-enhanced medical products alongside general-purpose LLMs are lacking in rheumatology.”

  • The authors state this as an assumption.

    This study is the first to compare the performance of a ChatGPT model with Prof. Valmed and OpenEvidence.

    “This study is, to our knowledge, the first to compare the (diagnostic) performance of a ChatGPT model with Prof. Valmed, a certified, subscription-based RAG-enhanced LLM medical product, and OpenEvidence, a freely accessible, medical RAG-based system.”
5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Freyer O, Jayabalan S, Kather JN et al (2025) Overcoming regulatory barriers to the implementation of AI agents in healthcare. Nat Med 31:3239–3243. https://doi.org/10.1038/s41591-025-03841-1 · DOI · Paper page

Cited by

Sorted by citations

Related

Authors
Kremer, P., Langballe, E., Haase, I., Bamberger, J., Kuhn, S., Krusche, M., & Knitza, J.
Affiliations
  • University Medical Center Hamburg-Eppendorf (DE)
  • Oslo University Hospital (NO)
  • Philipps University of Marburg (DE)
Corresponding author
Knitza, Johannes
ORCID
Venue
Rheumatology International
Publisher
Springer Science and Business Media LLC
Published
2026-01-10
Volume
46
Issue
1
Pages
31-31
ISSN
  • 0172-8172 (ISSN-L)
  • 1437-160X
DOI
10.1007/s00296-025-06068-y
PMC
PMC12790495
OpenAlex
W7119716229
Cited by
17
References
28
Citations by year
2026: 17
FWCI
33.83
Type
journal article
Language
en
Open access
Licence
Locations
  • Rheumatology International (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
Funders
Philipps-Universität Marburg
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
large language models, artificial intelligence in healthcare, diagnostic accuracy, differential diagnosis, OpenEvidence
MeSH terms
Large Language Models, Diagnosis, Differential, Humans, Predictive Value of Tests, Rheumatic Diseases, Rheumatology, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-30.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-30.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review